{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Training with Back-propagation Algorithms\n",
    "\n",
    "[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brainpy/brainpy/blob/master/docs_version2/tutorial_training/bp_training.ipynb)\n",
    "[![Open in Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/brainpy/brainpy/blob/master/docs_version2/tutorial_training/bp_training.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Back-propagation (BP) trainings have become foundations in machine learning algorithms. In this section, we are going to talk about how to train models with BP."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2025-10-06T05:18:57.739170Z",
     "start_time": "2025-10-06T05:18:52.657032Z"
    }
   },
   "source": [
    "import brainpy as bp\n",
    "import brainpy.math as bm\n",
    "import brainpy_datasets as bd\n",
    "import numpy as np\n",
    "\n",
    "bm.set_mode(bm.training_mode)  # set training mode, the models will compute with the training mode\n",
    "bm.set_platform('cpu')\n",
    "\n",
    "bp.__version__"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.0.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Here, we train two kinds of models to classify MNIST dataset. The first is ANN models commonly used in deep neural networks. The second is SNN models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Train a ANN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We first build a three layer ANN model:\n",
    "\n",
    "```bash\n",
    "\n",
    "i >> r >> o\n",
    "```\n",
    "\n",
    "where the recurrent layer ``r`` is a LSTM cell, the output ``o`` is a linear readout."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2025-10-06T05:18:57.903521Z",
     "start_time": "2025-10-06T05:18:57.898691Z"
    }
   },
   "source": [
    "class ANNModel(bp.DynamicalSystem):\n",
    "    def __init__(self, num_in, num_rec, num_out):\n",
    "        super(ANNModel, self).__init__()\n",
    "        self.rec = bp.dyn.LSTMCell(num_in, num_rec)\n",
    "        self.out = bp.dnn.Dense(num_rec, num_out)\n",
    "\n",
    "    def update(self, x):\n",
    "        return x >> self.rec >> self.out"
   ],
   "outputs": [],
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Before training this model, we get and clean the data we want."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2025-10-06T05:18:58.039946Z",
     "start_time": "2025-10-06T05:18:57.924620Z"
    }
   },
   "source": [
    "root = r\"D:\\data\"\n",
    "train_dataset = bd.vision.FashionMNIST(root, split='train', download=True)\n",
    "test_dataset = bd.vision.FashionMNIST(root, split='test', download=True)\n",
    "\n",
    "\n",
    "def get_data(dataset, batch_size=256):\n",
    "  rng = bm.random.default_rng()\n",
    "\n",
    "  def data_generator():\n",
    "    X = bm.array(dataset.data, dtype=bm.float_) / 255\n",
    "    Y = bm.array(dataset.targets, dtype=bm.float_)\n",
    "    key = rng.split_key()\n",
    "    rng.shuffle(X, key=key)\n",
    "    rng.shuffle(Y, key=key)\n",
    "    for i in range(0, len(dataset), batch_size):\n",
    "      yield X[i: i + batch_size], Y[i: i + batch_size]\n",
    "\n",
    "  return data_generator"
   ],
   "outputs": [],
   "execution_count": 3
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Then, we start to train our defined ANN model with ``brainpy.train.BPTT`` training interface."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2025-10-06T05:19:00.821153Z",
     "start_time": "2025-10-06T05:18:58.042956Z"
    }
   },
   "source": [
    "# model\n",
    "model = ANNModel(28, 100, 10)\n",
    "\n",
    "# loss function\n",
    "def loss_fun(predicts, targets):\n",
    "    predicts = bm.max(predicts, axis=1)\n",
    "    loss = bp.losses.cross_entropy_loss(predicts, targets)\n",
    "    acc = bm.mean(predicts.argmax(axis=-1) == targets)\n",
    "    return loss, {'acc': acc}\n",
    "\n",
    "# optimizer\n",
    "optimizer=bp.optim.Adam(lr=1e-3)\n",
    "\n",
    "# trainer\n",
    "trainer = bp.BPTT(model,\n",
    "                 loss_fun=loss_fun,\n",
    "                 loss_has_aux=True,\n",
    "                 optimizer=optimizer)"
   ],
   "outputs": [],
   "execution_count": 4
  },
  {
   "cell_type": "code",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "is_executing": true
    },
    "ExecuteTime": {
     "start_time": "2025-10-06T05:19:03.847081Z"
    }
   },
   "source": [
    "trainer.fit(train_data=get_data(train_dataset, 256),\n",
    "            test_data=get_data(test_dataset, 512),\n",
    "            num_epoch=10)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T08:26:35.728528700Z",
     "start_time": "2024-01-13T08:26:35.098672100Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABXNElEQVR4nO3deXhTdd428DtJm6RrutGNrpSllEJX6CZuOBVQR3ABl8FlVB6eUR+RV2dExBFcOuqMA+iUkRFkHBWrIsiMIBRHBQEFsUVk37ullLY06Zq0yXn/SBsampa2aXuy3J/rykVzcs7J94gzvf2tEkEQBBARERG5EKnYBRARERENNQYgIiIicjkMQERERORyGICIiIjI5TAAERERkcthACIiIiKXwwBERERELsdN7ALskdFoREVFBXx8fCCRSMQuh4iIiHpBEATU19cjPDwcUmnPbTwMQFZUVFQgMjJS7DKIiIioH0pLSxEREdHjOQxAVvj4+AAw/QP09fUVuRoiIiLqDa1Wi8jISPPv8Z4wAFnR0e3l6+vLAERERORgejN8hYOgiYiIyOUwABEREZHLYQAiIiIil8MxQEREREPMYDCgtbVV7DIcklwuv+IU994QPQDl5+fj9ddfh1qtxrhx47Bs2TJMnjzZ6rkPPPAA/vnPf3Y5npCQgEOHDgEA1q5diwcffLDLOc3NzVAqlQNbPBERUR8IgoDKykrU1dWJXYrDkkqliI2NhVwut+k+ogaggoICzJ8/H/n5+cjJycHbb7+NadOm4fDhw4iKiupy/vLly/GnP/3J/L6trQ1JSUm48847Lc7z9fXFsWPHLI4x/BARkdg6wk9wcDA8PT252G4fdSxUrFarERUVZdM/P1ED0BtvvIGHHnoIDz/8MABg2bJl2Lp1K1auXIm8vLwu56tUKqhUKvP7jRs34uLFi11afCQSCUJDQ3tdh06ng06nM7/XarV9fRQiIqIeGQwGc/gJDAwUuxyHNWzYMFRUVKCtrQ3u7u79vo9og6D1ej3279+P3Nxci+O5ubnYvXt3r+6xevVq3HDDDYiOjrY43tDQgOjoaERERODmm29GUVFRj/fJy8szhyuVSsVVoImIaMB1jPnx9PQUuRLH1tH1ZTAYbLqPaAGouroaBoMBISEhFsdDQkJQWVl5xevVajW2bNlibj3qEB8fj7Vr12LTpk1Yt24dlEolcnJycOLEiW7vtXDhQmg0GvOrtLS0fw9FRER0Bez2ss1A/fMTfRD05Q8iCEKvHm7t2rXw8/PDjBkzLI5nZmYiMzPT/D4nJwepqal48803sWLFCqv3UigUUCgUfS+eiIiIHJJoLUBBQUGQyWRdWnuqqqq6tApdThAErFmzBnPmzLniKHCpVIqJEyf22AJERERErkW0ACSXy5GWlobCwkKL44WFhcjOzu7x2m+//RYnT57EQw89dMXvEQQBxcXFCAsLs6leIiIiVyUIAubOnYuAgABIJBL4+flh/vz5YpdlE1G7wBYsWIA5c+YgPT0dWVlZWLVqFUpKSjBv3jwAprE55eXleO+99yyuW716NTIyMpCYmNjlnkuWLEFmZiZGjRoFrVaLFStWoLi4GH/729+G5JmupLZRj+oGHUaHXHmnWiIiInvw5ZdfYu3atfjmm28wYsQISKVSeHh4mD+PiYnB/PnzHSoUiRqAZs+ejZqaGixduhRqtRqJiYnYvHmzeVaXWq1GSUmJxTUajQbr16/H8uXLrd6zrq4Oc+fORWVlJVQqFVJSUrBjxw5MmjRp0J/nSgoPn8cj7/2I8cNV+PfjV4ldDhERUa+cOnUKYWFhV+yhcSQSQRAEsYuwN1qtFiqVChqNBr6+vgN239LaJkx+7Wu4SSX4ZcmNULrLBuzeRERk31paWnDmzBnExsZCqVRCEAQ0t9o2lbu/PNxlvZ5NdfkuDNHR0YiJiUFycjKWLVuGa6+9Ft9++63FNYMZLS7/59hZX35/iz4LzJVE+HsgyFuO6gY9DlVokRbtL3ZJREQkkuZWAxKe3yrKdx9eeiM85b2LAMuXL0dcXBxWrVqFffv2QSaTWezA8NlnnyEpKQlz587FI488MlglDzjuBj+EJBIJkiL8AADFpXWi1kJERNQbKpUKPj4+kMlkCA0NxbBhwyw+DwgIgEwmg4+PD0JDQ/u0E4OY2AI0xJIj/fDV0SocYAAiInJpHu4yHF56o2jf7eoYgIZYUqQfALYAERG5OolE0utuKBp47AIbYh0BqKS2CbWNenGLISIiGgByudzmvbmGGgPQEFN5uGPEMC8AYDcYERE5hZiYGOzYsQPl5eWorq4Wu5xeYQASQTIHQhMRkRNZunQpzp49i7i4uC6DpO0VA5AIkqP8ADAAERGRY5g/fz7Onj1rfv/NN99g2bJl5veZmZk4cOAAWlpaBnUNoIHEACSCjqnwB8rqHOZfFCIiImfCACSCsWG+kMukqGtqxbmaJrHLISIicjkMQCKQu0mREG5aovtAWZ24xRAREbkgBiCRJLdPhy8qqRO1DiIiIlfEACSSjgDEFiAiIqKhxwAkko4AdKhCC32bUdxiiIiIXAwDkEiiAz3h5+kOfZsRRyu1YpdDRETkUhiARMKd4YmIiMTDACQiboxKRESuJiYmxmIRRbFwG1oRpTAAERGRA7j22muRnJw8IMFl37598PLysr0oG7EFSEQTIlQAgNMXGqFpbhW5GiIiov4RBAFtbW29OnfYsGHw9PQc5IqujAFIRIHeCkQFmP4l+JnT4YmIyA498MAD+Pbbb7F8+XJIJBJIJBKsXbsWEokEW7duRXp6OhQKBXbu3IlTp07h1ltvRUhICLy9vTFx4kRs377d4n6Xd4FJJBK88847mDlzJjw9PTFq1Chs2rRp0J+LAUhkHeOADrAbjIjItQgCoG8U59WHfSiXL1+OrKwsPPLII1Cr1VCr1YiMjAQA/P73v0deXh6OHDmCCRMmoKGhAdOnT8f27dtRVFSEG2+8EbfccgtKSkp6/I4lS5Zg1qxZ+PnnnzF9+nTce++9qK2ttekf75VwDJDIkiP98O8DFRwHRETkalqbgFfCxfnuZysAee/G4ahUKsjlcnh6eiI0NBQAcPToUQDA0qVL8atf/cp8bmBgIJKSkszvX3rpJWzYsAGbNm3CY4891u13PPDAA7j77rsBAK+88grefPNN7N27F1OnTu3zo/UWW4BElhxpGgdUXKrhzvBERORQ0tPTLd43Njbi97//PRISEuDn5wdvb28cPXr0ii1AEyZMMP/s5eUFHx8fVFVVDUrNHdgCJLJx4Sq4SSWobtChvK4ZEf7iDwwjIqIh4O5paokR67sHwOWzuZ5++mls3boVf/7znzFy5Eh4eHjgjjvugF6v77kcd3eL9xKJBEbj4O6SwAAkMqW7DPFhPvilXIsDpRoGICIiVyGR9LobSmxyuRwGg+GK5+3cuRMPPPAAZs6cCQBoaGjA2bNnB7m6/mEXmB1INq8HdFHcQoiIiKyIiYnBDz/8gLNnz6K6urrb1pmRI0fis88+Q3FxMQ4cOIB77rln0Fty+osByA50bIlxoFQjbiFERERWPPXUU5DJZEhISMCwYcO6HdPz17/+Ff7+/sjOzsYtt9yCG2+8EampqUNcbe9IBI687UKr1UKlUkGj0cDX13fQv+9kVT1ueGMHPNxlOPhCLtxkzKVERM6mpaUFZ86cQWxsLJRKpdjlOKye/jn25fc3f9PagRFB3vBRuKG51YBj5+vFLoeIiMjpMQDZAalUggnt0+HZDUZERDT4GIDsRMc4IA6EJiIiGnwMQHYi2bwlBluAiIiIBhsDkJ3oCEDHq+rRoOvdjrpEROR4OPfINgP1z48ByE4E+yoRrlJCEICDZWwFIiJyNh2rHTc1NYlciWPrWFVaJpPZdB+uBG1HkqP8UHGwEsWldciKCxS7HCIiGkAymQx+fn7mPa48PT0hkUhErsqxGI1GXLhwAZ6ennBzsy3CMADZkaQIP2w+WIkD3BmeiMgpdeymPtgbfTozqVSKqKgom8Oj6AEoPz8fr7/+OtRqNcaNG4dly5Zh8uTJVs994IEH8M9//rPL8YSEBBw6dMj8fv369Vi8eDFOnTqFuLg4vPzyy+Z9SezZpS0x6kStg4iIBodEIkFYWBiCg4PR2toqdjkOSS6XQyq1fQSPqAGooKAA8+fPR35+PnJycvD2229j2rRpOHz4MKKiorqcv3z5cvzpT38yv29ra0NSUhLuvPNO87E9e/Zg9uzZePHFFzFz5kxs2LABs2bNwnfffYeMjIwhea7+ShyuglQCVGpbUKlpQaiKK4USETkjmUxm8xgWso2oW2FkZGQgNTUVK1euNB8bO3YsZsyYgby8vCtev3HjRtx22204c+YMoqOjAQCzZ8+GVqvFli1bzOdNnToV/v7+WLduXa/qGuqtMDqbumwHjlbW4++/ScPUxNAh/W4iIiJH5hBbYej1euzfvx+5ubkWx3Nzc7F79+5e3WP16tW44YYbzOEHMLUAXX7PG2+8scd76nQ6aLVai5dYzOsBldWJVgMREZGzEy0AVVdXw2AwICQkxOJ4SEgIKisrr3i9Wq3Gli1b8PDDD1scr6ys7PM98/LyoFKpzK/IyMg+PMnAMo8DKqkTrQYiIiJnJ/o6QJeP4hYEoVcju9euXQs/Pz/MmDHD5nsuXLgQGo3G/CotLe1d8YMgqT0AHSzXwGDkYllERESDQbRB0EFBQZDJZF1aZqqqqrq04FxOEASsWbMGc+bMgVwut/gsNDS0z/dUKBRQKBR9fILBMTrEB55yGRp0bTh1oQGjQ3zELomIiMjpiNYCJJfLkZaWhsLCQovjhYWFyM7O7vHab7/9FidPnsRDDz3U5bOsrKwu99y2bdsV72kvZFIJEoebdobndHgiIqLBIWoX2IIFC/DOO+9gzZo1OHLkCJ588kmUlJRg3rx5AExdU/fdd1+X61avXo2MjAwkJiZ2+eyJJ57Atm3b8Oqrr+Lo0aN49dVXsX37dsyfP3+wH2fApHA9ICIiokEl6jpAs2fPRk1NDZYuXQq1Wo3ExERs3rzZPKtLrVajpKTE4hqNRoP169dj+fLlVu+ZnZ2Njz76CM899xwWL16MuLg4FBQU2P0aQJ0lmXeGrxO1DiIiImcl6jpA9krMdYAAoKKuGdl/+i9kUgl+eeFGeMi5WBYREdGVOMQ6QNS9MJUSw3wUMBgFHKrgzvBEREQDjQHIDkkkEu4LRkRENIgYgOwUAxAREdHgYQCyUwxAREREg4cByE6Nj1BBIgHKLjajukEndjlEREROhQHITvkq3RE3zBsAp8MTERENNAYgO5YU4QeAAYiIiGigMQDZseQoPwBAEQMQERHRgGIAsmPJnVqAuF4lERHRwGEAsmPxYT6Qu0mhbWnDmepGscshIiJyGgxAdsxdJkViuGkp7wNldeIWQ0RE5EQYgOxccqQ/AKC4pE7cQoiIiJwIA5CdS4pUAQCKy7gnGBER0UBhALJzKe0tQEcqtNC1GUSuhoiIyDkwANm5yAAPBHjJoTcYcbhCK3Y5REREToEByM5JJBIkRZi6wbggIhER0cBgAHIA5oHQDEBEREQDggHIAXQMhD7AgdBEREQDggHIASRH+gEAzlQ3oq5JL24xREREToAByAH4ecoRE+gJgK1AREREA4EByEF0tAJxQUQiIiLbMQA5iKT2AMQtMYiIiGzHAOQgzC1A3BmeiIjIZgxADmJsmC/cZRLUNupRdrFZ7HKIiIgcGgOQg1C6y5AQZtoZvojrAREREdmEAciBmMcBMQARERHZhAHIgXQeB0RERET9xwDkQDpagH4p16DVYBS3GCIiIgfGAORAYgO94Kt0g67NiGOV9WKXQ0RE5LAYgByIVCoxtwKxG4yIiKj/GIAcDMcBERER2Y4ByMEkRfgB4EwwIiIiWzAAOZiOLrCTFxpQ39IqbjFEREQOigHIwQzzUWC4nwcEATjIneGJiIj6hQHIASVH+QHgitBERET9xQDkgJI5DoiIiMgmDEAOqKMFiDvDExER9Y/oASg/Px+xsbFQKpVIS0vDzp07ezxfp9Nh0aJFiI6OhkKhQFxcHNasWWP+fO3atZBIJF1eLS0tg/0oQyYxXAWZVIKqeh0qtc7zXEREREPFTcwvLygowPz585Gfn4+cnBy8/fbbmDZtGg4fPoyoqCir18yaNQvnz5/H6tWrMXLkSFRVVaGtrc3iHF9fXxw7dszimFKpHLTnGGoechnGhPjgsFqL4pI6hI33ELskIiIihyJqAHrjjTfw0EMP4eGHHwYALFu2DFu3bsXKlSuRl5fX5fwvv/wS3377LU6fPo2AgAAAQExMTJfzJBIJQkNDe12HTqeDTqczv9dqtX18kqGXFOlnCkBldZg2PkzscoiIiByKaF1ger0e+/fvR25ursXx3Nxc7N692+o1mzZtQnp6Ol577TUMHz4co0ePxlNPPYXm5maL8xoaGhAdHY2IiAjcfPPNKCoq6rGWvLw8qFQq8ysyMtK2hxsCKR0rQpfUiVoHERGRIxItAFVXV8NgMCAkJMTieEhICCorK61ec/r0aXz33Xf45ZdfsGHDBixbtgyffvopHn30UfM58fHxWLt2LTZt2oR169ZBqVQiJycHJ06c6LaWhQsXQqPRmF+lpaUD85CDqGNBxIPlGhiMHAhNRETUF6J2gQGm7qrOBEHocqyD0WiERCLBBx98AJVKBcDUjXbHHXfgb3/7Gzw8PJCZmYnMzEzzNTk5OUhNTcWbb76JFStWWL2vQqGAQqEYoCcaGiODveEll6FRb8CJqnrEh/qKXRIREZHDEK0FKCgoCDKZrEtrT1VVVZdWoQ5hYWEYPny4OfwAwNixYyEIAsrKyqxeI5VKMXHixB5bgByRTCrB+AjTPweuB0RERNQ3ogUguVyOtLQ0FBYWWhwvLCxEdna21WtycnJQUVGBhoYG87Hjx49DKpUiIiLC6jWCIKC4uBhhYc43UDg50h8Ad4YnIiLqK1HXAVqwYAHeeecdrFmzBkeOHMGTTz6JkpISzJs3D4BpbM59991nPv+ee+5BYGAgHnzwQRw+fBg7duzA008/jd/+9rfw8DBNBV+yZAm2bt2K06dPo7i4GA899BCKi4vN93Qmye3jgIo4EJqIiKhPRB0DNHv2bNTU1GDp0qVQq9VITEzE5s2bER0dDQBQq9UoKSkxn+/t7Y3CwkI8/vjjSE9PR2BgIGbNmoWXXnrJfE5dXR3mzp2LyspKqFQqpKSkYMeOHZg0adKQP99g6whAx8/Xo0nfBk+56EO6iIiIHIJE4F4KXWi1WqhUKmg0Gvj62vfg4sxXvkKltgUFczORMSJQ7HKIiIhE05ff36JvhUG2SYpsHwhdViduIURERA6EAcjBcSA0ERFR3zEAOThzC1CpRuRKiIiIHAcDkIObEOEHiQQor2tGVT13hiciIuoNBiAH561ww6hgbwBsBSIiIuotBiAn0DEdvrj0oriFEBEROQgGICfQsTEqW4CIiIh6hwHICSSbA1AdjNwZnoiI6IoYgJzAmBAfKN2lqNe14XR1o9jlEBER2T0GICfgJpNi/HDTdHiuB0RERHRlDEBOIinCD4CpG4yIiIh6xgDkJJKj/ACwBYiIiKg3GICcREcL0BG1Fi2tBnGLISIisnMMQE4iwt8DQd5ytBkFHKrQil0OERGRXWMAchISiYTjgIiIiHqJAciJXFoRuk7UOoiIiOwdA5ATMa8IXVYnah1ERET2jgHIiXR0gZ2raUJto17cYoiIiOwYA5ATUXm6Y0SQFwC2AhEREfWEAcjJmMcBldSJWgcREZE9YwByMhwHREREdGUMQE6m887wgsCd4YmIiKxhAHIy8WE+kMukuNjUipLaJrHLISIisksMQE5G4SZDQrgvAK4HRERE1B0GICfEBRGJiIh6xgDkhBiAiIiIesYA5IQ6ZoIdqtBC32YUtxgiIiI7xADkhGICPeHn6Q59mxFHK7kzPBER0eUYgJwQd4YnIiLqGQOQk+roBitiACIiIuqCAchJpXAgNBERUbcYgJzUhAgVAOD0hUZomltFroaIiMi+MAA5qUBvBaICPAEAP3NfMCIiIgsMQE4sqdO+YERERHQJA5AT44KIRERE1jEAObHkSNM4oOJSDXeGJyIi6kT0AJSfn4/Y2FgolUqkpaVh586dPZ6v0+mwaNEiREdHQ6FQIC4uDmvWrLE4Z/369UhISIBCoUBCQgI2bNgwmI9gt8aFq+AmlaC6QYfyumaxyyEiIrIbogaggoICzJ8/H4sWLUJRUREmT56MadOmoaSkpNtrZs2aha+++gqrV6/GsWPHsG7dOsTHx5s/37NnD2bPno05c+bgwIEDmDNnDmbNmoUffvhhKB7JrijdZYgP8wEAHCjViFwNERGR/ZAIIvaNZGRkIDU1FStXrjQfGzt2LGbMmIG8vLwu53/55Ze46667cPr0aQQEBFi95+zZs6HVarFlyxbzsalTp8Lf3x/r1q3rVV1arRYqlQoajQa+vr59fCr78tzGg3j/+xI8MjkWi25KELscIiKiQdOX39+itQDp9Xrs378fubm5Fsdzc3Oxe/duq9ds2rQJ6enpeO211zB8+HCMHj0aTz31FJqbL3Xv7Nmzp8s9b7zxxm7vCZi61bRarcXLWVzaEoMtQERERB3cxPri6upqGAwGhISEWBwPCQlBZWWl1WtOnz6N7777DkqlEhs2bEB1dTV+97vfoba21jwOqLKysk/3BIC8vDwsWbLExieyTylRfgCAg+UatBmMcJOJPuyLiIhIdKL/NpRIJBbvBUHocqyD0WiERCLBBx98gEmTJmH69Ol44403sHbtWotWoL7cEwAWLlwIjUZjfpWWltrwRPZlRJA3fBRuaG414Pj5BrHLISIisguiBaCgoCDIZLIuLTNVVVVdWnA6hIWFYfjw4VCpVOZjY8eOhSAIKCsrAwCEhob26Z4AoFAo4Ovra/FyFlKpBBPM0+HrxC2GiIjITogWgORyOdLS0lBYWGhxvLCwENnZ2VavycnJQUVFBRoaLrVkHD9+HFKpFBEREQCArKysLvfctm1bt/d0BZfGAdWJWgcREZG9ELULbMGCBXjnnXewZs0aHDlyBE8++SRKSkowb948AKauqfvuu898/j333IPAwEA8+OCDOHz4MHbs2IGnn34av/3tb+Hh4QEAeOKJJ7Bt2za8+uqrOHr0KF599VVs374d8+fPF+MR7QJXhCYiIrIk2iBowDRlvaamBkuXLoVarUZiYiI2b96M6OhoAIBarbZYE8jb2xuFhYV4/PHHkZ6ejsDAQMyaNQsvvfSS+Zzs7Gx89NFHeO6557B48WLExcWhoKAAGRkZQ/589qIjAB2vqkeDrg3eClH/2omIiEQn6jpA9sqZ1gHqkJ33FSo0LVj3SCay4gLFLoeIiGjAOcQ6QDS0zDvDl9WJWgcREZE9YAByEeZxQCV1otZBRERkDxiAXARbgIiIiC5hAHIR44erIJUAak0LzmtbxC6HiIhIVAxALsJL4YbRIaad4TkdnoiIXB0DkAvhekBEREQmDEAuxDwOiAGIiIhcHAOQC+loAfq5TAODkcs/ERGR62IAciGjgr3h4S5Dg64Npy9wZ3giInJdDEAuxE0mxfgI087wRewGIyIiF8YA5GJSOA6IiIiIAcjVJHEmGBEREQOQq+kYCH20sh4trQZxiyEiIhIJA5CLCVMpMcxHAYNRwC/lGrHLISIiEkW/AlBpaSnKysrM7/fu3Yv58+dj1apVA1YYDQ6JRMIFEYmIyOX1KwDdc889+PrrrwEAlZWV+NWvfoW9e/fi2WefxdKlSwe0QBp4DEBEROTq+hWAfvnlF0yaNAkA8PHHHyMxMRG7d+/Ghx9+iLVr1w5kfTQIGICIiMjV9SsAtba2QqFQAAC2b9+OX//61wCA+Ph4qNXqgauOBsX4CBUkEqDsYjOqG3Ril0NERDTk+hWAxo0bh7///e/YuXMnCgsLMXXqVABARUUFAgMDB7RAGni+SnfEDfMGwPWAiIjINfUrAL366qt4++23ce211+Luu+9GUlISAGDTpk3mrjGyb0kRfgAYgIiIyDW59eeia6+9FtXV1dBqtfD39zcfnzt3Ljw9PQesOBo8yVF+WP9TGbfEICIil9SvFqDm5mbodDpz+Dl37hyWLVuGY8eOITg4eEALpMGR3KkFSBC4MzwREbmWfgWgW2+9Fe+99x4AoK6uDhkZGfjLX/6CGTNmYOXKlQNaIA2O+DAfyN2k0La04Ux1o9jlEBERDal+BaCffvoJkydPBgB8+umnCAkJwblz5/Dee+9hxYoVA1ogDQ53mRSJ4b4AgANldeIWQ0RENMT6FYCamprg4+MDANi2bRtuu+02SKVSZGZm4ty5cwNaIA2e5EhTF2ZxSZ24hRAREQ2xfgWgkSNHYuPGjSgtLcXWrVuRm5sLAKiqqoKvr++AFkiDJylSBQAoLuOeYERE5Fr6FYCef/55PPXUU4iJicGkSZOQlZUFwNQalJKSMqAF0uBJaW8BOlKhha6NO8MTEZHr6Nc0+DvuuANXXXUV1Gq1eQ0gAJgyZQpmzpw5YMXR4IoM8ECAlxy1jXocUdebt8ggIiJydv1qAQKA0NBQpKSkoKKiAuXl5QCASZMmIT4+fsCKo8ElkUiQFNHeDVZyUeRqiIiIhk6/ApDRaMTSpUuhUqkQHR2NqKgo+Pn54cUXX4TRaBzoGmkQJbW3+hzgOCAiInIh/eoCW7RoEVavXo0//elPyMnJgSAI2LVrF1544QW0tLTg5ZdfHug6aZBwZ3giInJF/QpA//znP/HOO++Yd4EHgKSkJAwfPhy/+93vGIB6Uv4T4BMG+IaJXQmAS3uCnaluRF2THn6ecnELIiIiGgL96gKrra21OtYnPj4etbW1NhfltI5vBd6dBnx0N6BvErsaAIC/lxwxgab929gNRkRErqJfASgpKQlvvfVWl+NvvfUWJkyYYHNRTitoFODuCVQUAZ8/CtjJHlzmcUDsBiMiIhfRry6w1157DTfddBO2b9+OrKwsSCQS7N69G6Wlpdi8efNA1+g8AkYAs98H3rsVOPQZMCweuPYPYleF5Eg/fF5cwXFARETkMvrVAnTNNdfg+PHjmDlzJurq6lBbW4vbbrsNhw4dwrvvvjvQNTqXmBzg5jdMP3/zCvDLZ+LWA8sWIO4MT0RErkAiDOBvvAMHDiA1NRUGg2OvKqzVaqFSqaDRaAZva4+ti4A9bwFuSuDBzcDwtMH5nl5oaTVg/Atb0WoQsPP31yEywFO0WoiIiPqrL7+/+70Q4kDJz89HbGwslEol0tLSsHPnzm7P/eabbyCRSLq8jh49aj5n7dq1Vs9paWkZisfpvV8tBUblAm0twLp7AG2FaKUo3WUYG2b6F4XdYERE5ApEDUAFBQWYP38+Fi1ahKKiIkyePBnTpk1DSUlJj9cdO3YMarXa/Bo1apTF576+vhafq9VqKJXKwXyUvpPKgNtXA8PGAg2VwLq7RJ0ZxvWAiIjIlYgagN544w089NBDePjhhzF27FgsW7YMkZGRWLlyZY/XBQcHIzQ01PySyWQWn0skEovPQ0NDe7yfTqeDVqu1eA0JpS9wz0eAZyCgPgBsnAeItJJ2MmeCERGRC+nTLLDbbrutx8/r6up6fS+9Xo/9+/fjmWeesTiem5uL3bt393htSkoKWlpakJCQgOeeew7XXXedxecNDQ2Ijo6GwWBAcnIyXnzxxR53qc/Ly8OSJUt6XfuA8o8BZn8A/PMW4PDnwDd5wPWLhryMjoHQB8s1aDUY4S4TvXeUiIho0PTpt5xKperxFR0djfvuu69X96qurobBYEBISIjF8ZCQEFRWVlq9JiwsDKtWrcL69evx2WefYcyYMZgyZQp27NhhPic+Ph5r167Fpk2bsG7dOiiVSuTk5ODEiRPd1rJw4UJoNBrzq7S0tFfPMGCis4Bblpt+3vEacPDTof1+ALGBXvBVukHXZsSxyvoh/34iIqKh1KcWoMGY4i6RSCzeC4LQ5ViHMWPGYMyYMeb3WVlZKC0txZ///GdcffXVAIDMzExkZmaaz8nJyUFqairefPNNrFixwup9FQoFFAqFrY9im5R7gepjwK7lwMbfmVqGItKH7OulUgmSIv2w80Q1ikvrkDhcNWTfTURENNRE6+cICgqCTCbr0tpTVVXVpVWoJ5mZmT227kilUkycOLHHc+zGlD8Co6cBBh3w0T2ApmxIv57jgIiIyFWIFoDkcjnS0tJQWFhocbywsBDZ2dm9vk9RURHCwrrfWFQQBBQXF/d4jt2QyoDb/wEEjwMazrfPDGscsq/v2BiVM8GIiMjZ9WsrjIGyYMECzJkzB+np6cjKysKqVatQUlKCefPmATCNzSkvL8d7770HAFi2bBliYmIwbtw46PV6vP/++1i/fj3Wr19vvueSJUuQmZmJUaNGQavVYsWKFSguLsbf/vY3UZ6xzxQ+pplh/7geqDwIfDYXmPUvQDr4WbVjIPTJCw2ob2mFj9J90L+TiIhIDKIGoNmzZ6OmpgZLly6FWq1GYmIiNm/ejOjoaACAWq22WBNIr9fjqaeeQnl5OTw8PDBu3Dh88cUXmD59uvmcuro6zJ07F5WVlVCpVEhJScGOHTswadKkIX++fvOLap8ZdjNw9D/A1y8BU54f9K8d5qPAcD8PlNc142CZBtkjgwb9O4mIiMQwoFthOIsh2QqjNw58BGz4H9PPM1cBSbMH/Ssf/fAnfPGzGk/fOAaPXjdy0L+PiIhooDjUVhjUg6S7gKsWmH7e9BhQunfQvzK5fRwQB0ITEZEzYwCyd9cvBuJvBgx608ywup63CbFVcpQfANNAaDYOEhGRs2IAsndSKTDzbSB0PNB4AVh3N6BrGLSvSwxXQSaVoKpeh0qtnW0gS0RENEAYgByBwhu4+yPAKxg4/wvw2SODtmeYh1yGMSE+AIDikrpB+Q4iIiKxMQA5ClUEcNeHgEwBHNsMfDV4e5d1TIcvLqsbtO8gIiISEwOQI4mcCNzavp7RrmVA8YeD8jUpHQGILUBEROSkGIAczYQ7gaufNv286f+Ac3sG/Cs67wxvMHIgNBEROR8GIEd07bPA2F8Dxlag4F7g4rkBvf3IYG94yWVo0htwooo7wxMRkfNhAHJEUikw8+9AWBLQVGPaM6xFO2C3l0klGB9h2g2e6wEREZEzYgByVHIv4K51gHcoUHUYWP8wYDQM2O2TI/0BcGNUIiJyTgxAjkw1HLj7Q8BNCZzYCmz/44DdOjnS1AJUXKoZsHsSERHZCwYgRzc8DZiRb/p595vAT/8akNt2tAAdq9SiSd82IPckIiKyFwxAziDxduCaZ0w//+dJ4Owum28ZqlIixFcBowD8Uj5w44uIiIjsAQOQs7jmD8C4me0zw34D1J6x+ZbJHesBlV60+V5ERET2hAHIWUilwK35QHgK0FzbPjPMtvE7HesBHeA4ICIicjIMQM5E7mmaGeYTBlw4Cnz6W8DQ//E7l1qA6gamPiIiIjvBAORsfMOAu9cBbh7Aye1A4eJ+32r8cBUkEqC8rhlV9dwZnoiInAcDkDMKTzEtlAgA3+cD+9f26zY+SneMCvYGwG4wIiJyLgxAzmrcDOC6Raafv/h/wJmd/bpNUoQfAK4ITUREzoUByJld/bRpiryxDfh4DlBzqs+3SI7yA8BxQERE5FwYgJyZRALc+jfTYonNF00zw5rr+nSLjoHQB8rqYOTO8ERE5CQYgJyduwdw14eA73Cg+jjw6YN9mhk2JsQHSncp6lvacLq6cRALJSIiGjoMQK7AJ9Q0M8zdEzj1X2Drs72+1E0mxfjh3BmeiIicCwOQqwhLAm5bZfp579vAvnd6fWnHQGiOAyIiImfBAORKxt4CTHne9PPm3wOnvu7VZR0DoQ+U1Q1OXUREREOMAcjVXLUAmDAbEAzAJ/cD1SeveElHC9ARtRYtrYZBLpCIiGjwMQC5GokEuGUFEDHRtFfYutmmGWI9iPD3QJC3HK0GAYfV3BmeiIgcHwOQK3JXts8MiwBqTgIf3w8YWrs9XSKRXBoHVFI3NDUSERENIgYgV+UdDNzzEeDuBZz5FvjymR5P77weEBERkaNjAHJloeOB298BIDHNCtv7j25PTeLO8ERE5EQYgFxd/HTghhdMP2/5A3DyK6undXSBnatpQm2jfmhqIyIiGiQMQATkPAEk3dM+M+xB4MLxLqeoPN0xIsgLALvBiIjI8TEAUfvMsGVAZCag0wAfzgKaaruc1jEOiAOhiYjI0TEAkYmbArjrA8AvCrh4Bvj4vi4zw5I4EJqIiJwEAxBd4hUE3F0AyL2BszuBzU8BwqUd4M0zwUrrIAjcGZ6IiBwXAxBZCkkAbl8NQALsXwv88Lb5o/gwH8hlUlxsakVJbZNoJRIREdlK9ACUn5+P2NhYKJVKpKWlYefOnd2e+80330AikXR5HT161OK89evXIyEhAQqFAgkJCdiwYcNgP4ZzGTMVyH3R9PPWhcCJ7QAAhZsMCeG+ADgdnoiIHJuoAaigoADz58/HokWLUFRUhMmTJ2PatGkoKSnp8bpjx45BrVabX6NGjTJ/tmfPHsyePRtz5szBgQMHMGfOHMyaNQs//PDDYD+Oc8l6DEj5DSAYgU8fBKpMITOZ6wEREZETkAgiDubIyMhAamoqVq5caT42duxYzJgxA3l5eV3O/+abb3Ddddfh4sWL8PPzs3rP2bNnQ6vVYsuWLeZjU6dOhb+/P9atW9erurRaLVQqFTQaDXx9ffv2UM6kTQ/8awZwbhfgHwM8/F9sPN6C+QXFSInyw4bf5YhdIRERkVlffn+L1gKk1+uxf/9+5ObmWhzPzc3F7t27e7w2JSUFYWFhmDJlCr7++muLz/bs2dPlnjfeeGOP99TpdNBqtRYvAuAmB2b9C/CLBi6eBT6eg6RwTwDAoQot9G1GcesjIiLqJ9ECUHV1NQwGA0JCQiyOh4SEoLKy0uo1YWFhWLVqFdavX4/PPvsMY8aMwZQpU7Bjxw7zOZWVlX26JwDk5eVBpVKZX5GRkTY8mZPxCgTu+RhQ+ALndiFmz3NQKd2gbzPiaCWDIhEROSbRB0FLJBKL94IgdDnWYcyYMXjkkUeQmpqKrKws5Ofn46abbsKf//znft8TABYuXAiNRmN+lZaW9vNpnFRwPHDHGkAihaT4ffzBz7RdxgGOAyIiIgclWgAKCgqCTCbr0jJTVVXVpQWnJ5mZmThx4oT5fWhoaJ/vqVAo4Ovra/Giy4z6FZD7MgDgrrp/4DppEYoYgIiIyEGJFoDkcjnS0tJQWFhocbywsBDZ2dm9vk9RURHCwsLM77Oysrrcc9u2bX26J3Uj83+B1PshhREr3N9C1YmfUKlpEbsqIiKiPnMT88sXLFiAOXPmID09HVlZWVi1ahVKSkowb948AKauqfLycrz33nsAgGXLliEmJgbjxo2DXq/H+++/j/Xr12P9+vXmez7xxBO4+uqr8eqrr+LWW2/F559/ju3bt+O7774T5RmdikQCTP8z2i6chE/pLqzW/x7fv/EvnE68BRlTfwOZT7DYFRIREfWKqAFo9uzZqKmpwdKlS6FWq5GYmIjNmzcjOjoaAKBWqy3WBNLr9XjqqadQXl4ODw8PjBs3Dl988QWmT59uPic7OxsfffQRnnvuOSxevBhxcXEoKChARkbGkD+fU3KTw+3u99G89jZ4VBXhGvwEHPoJxkNL0RSaDs8JM4D4m4CAWLErJSIi6pao6wDZK64D1AuCAEPVUfxc+D7cT2xBouSU5efB44CxN5vCUOgEU+sRERHRIOrL728GICsYgPrmvLYFKz77GrITW5Ar/RGZsiNwQ6c1glRRpiA09mYgMhOQidrwSERETooByEYMQP3z1ZHzeP7zQ2iou4DrpUW4P+AXJLX8CElb86WTPAKAMdNNgSjuOsDdQ7yCiYjIqTAA2YgBqP8adW34a+FxrNl1BkYBGKY04M+ptbja8AMkx7cAzRcvnezuCYycAsTfDIy+EfDwF69wIiJyeAxANmIAst0v5Ros/OwgDpZrAABp0f545daxGKP7BTj6H+DoF4Cm04KTUjcgOgcYe4uphUg1XKTKiYjIUTEA2YgBaGAYjALe23MWf956DI16A9ykEvzPNSPw+PWjoHSTApU/A0faw1DVIcuLw1Pbxw3dAgSN5iBqIiK6IgYgGzEADayKumb8cdMhFB4+DwCIDvTEyzPG46pRQZdOqjkFHNtsCkSlPwDo9K9l4EhTN1n8zcDwNEAq+g4uRERkhxiAbMQANDi2HqrEHz8/hEqtafXomSnDseimsQjyVlie2FBlCkNHvwBOfwMY9Jc+8w4F4tsHUcdcbdqxnoiICAxANmMAGjz1La34y7bj+OeesxAEwM/THc9OG4s70yOsb1jbogVObjeFoRPbAF2nHegVvsCoXNP0+pE3AAqfoXsQIiKyOwxANmIAGnwHSuuw8LODOKw2BZpJsQF4ZeZ4jAz27v6iNh1wdqepm+zYZqDh/KXPZApgxLWmlqEx0wHvYYP7AEREZHcYgGzEADQ02gxGvLvrLN4oPI7mVgPkMin+99o4/O+1cVC6y3q+2GgEyn80zSg78h+gtvNK1BIgKrN93BC35SAichUMQDZiABpapbVNeP7zX/D1sQsAgBFBXnhpZiKy44KucGU7QQAuHAOO/tvUVVZRZPk5t+UgInIJDEA2YgAaeoIgYPPBSrzw70O4UK8DANyRFoFF08fC36uPA501ZcDRzabWobPfAYLh0mfcloOIyGkxANmIAUg82pZWvPblUXzwQwkEAQjwkmPR9LG4LXW49UHSV9JUaxo8feTfwMmvAGvbcoz6FRA5CfANH7gHISKiIccAZCMGIPHtP3cRz352EMfO1wMAckYG4qUZ4xEb5NX/m+qbgNNfm7rJjm223JYDAHyHAxHpQMRE0yssiXuVERE5EAYgGzEA2YdWgxH/2Hkay7efgK7NCLmbFI9fNxL/c00c5G42LoZoaANK9rR3k+0yrUQtGC3PkboBoeMvBaKIiYB/DMcQERHZKQYgGzEA2ZeSmiYs2ngQO09UAwBGBnsj77bxmBgTMHBfomswDZ4u2weU/QiU7QUaL3Q9zzOoPQy1txQNT+X6Q0REdoIByEYMQPZHEARsOlCBF/9zGNUNppWh754UiWemjoXK030wvhCoK+kUiPYB6gOAsfWyEyVAcIJl11nQaG7XQUQkAgYgGzEA2a+6Jj3+tOUoPtpn2kk+yFuOxTcn4NdJ4f0bJN0XrS1A5cH2UNQejDQlXc9T+Jr2LDN3naUDngPYWkVERFYxANmIAcj+7T1Ti2c3HMTJqgYAwNWjh+HlGYmIDPAc2kLqKy+1EJX9CFT8BLQ2dT0vIM6y6yxkHCAbhJYrIiIXxgBkIwYgx6BrM2DVt6fx5tcnoW8zQukuxRNTRuPhybFwl4nUBWVoA6oOW44lqjnZ9Tw3DyA8xbLrzDds6OslInIiDEA2YgByLKcvNOC5jb9g96kaAEB8qA9euW08UqP8Ra6sXVMtUL6/U9fZfkCn6Xqeb4SVafjKoa+XiMhBMQDZiAHI8QiCgM9+KsdLXxzGxaZWSCTAbzKi8fTUMfBV2llXk9EI1JywHEtUddjKNHz3y6bhp3MaPhFRDxiAbMQA5LhqG/V4ZfMRfLq/DAAQ7KPAC78eh2mJoYM/SNoWuvrLpuHvu/I0/MhJpm40TsMnIgLAAGQzBiDHt/tUNRZt+AVnqhsBANfHB2PpreMQ4T/Eg6T7SxCAunOdBljvA9Q/d52GL5FaTsOPzAQC49hKREQuiQHIRgxAzqGl1YD8b05h5Tcn0WoQ4OEuw//LHY0HsmPgJtYgaVu0tgCVP182Db+063meQUBkBhCVYQpE4cmAm2LIyyUiGmoMQDZiAHIuJ6vq8exnv2Dv2VoAwLhwX+TdNh4TIvzELWwgaNVA+Y9A6V7Tq6IIMOgsz5EpTF1lHYEoMgPwChSnXiKiQcQAZCMGIOdjNAr4ZH8pXtl8FJrmVkglwP3ZMfh/uWPgrXATu7yB06YDKoqB0u+Bkh+A0h+Apuqu5wWOuhSIojKBwJHsNiMih8cAZCMGIOdV3aDDS/85jI3FFQCAMJUSz04fi6mJoeKtHTSYBAGoOdUeiL43BaLq413P8ww0tQxFZpgCUVgyp+ATkcNhALIRA5Dz23niAp7b+AvO1ZhWbQ7yluO21AjMSo/AyGAnn1XVVNveZdbeSlS+30q3mdzUbdYRiCIzAK8gceolIuolBiAbMQC5hpZWA1Z+cwof/FCC6oZLASA1yg+z0iNx04Qw+NjbGkKDoU1v2ui1cyuRtSn4gSPbu8zau86CRrHbjIjsCgOQjRiAXEurwYhvjl3Axz+W4r9Hq2Awmv4n4eEuw00TwjArPRITY/ztex2hgSQIQO1pUxDqCEQXjnY9zyPgstlmKew2IyJRMQDZiAHIdVXVt2DDT+X4+MdSnLrQaD4eG+SFO9MjcHtqBEJ8XfCXfFOtaep9RyAq3w+0tVieI5Obxg51nm3mPUyUconINTEA2YgBiARBwE8ldfh4Xyn+83MFGvUGAIBUAlw7Jhiz0iNxfXww5G5OOHC6N9r0pjWJSr6/NJaosarreQFxl8YQRWWaZp9JXfSfGRENOgYgGzEAUWeNujZsPqjGxz+WYt/Zi+bjgV5yzEwZjlkTIzE6xMkHTl+JIAAXz7RPvW8PRBeOdD3Pw99ytll4CuDuMfT1EpFTYgCyEQMQdef0hQZ8sr8M6/eXoar+0sDp5EjTwOlbklxk4HRvNF8ESvdZzjZra7Y8R+puWqm682wz72BRyiUix8cAZCMGILqSNoMR3x43DZz+6kgV2toHTivdpZg+3jRwOiM2wHUGTvdGmx6oPGg526zhfNfzVFGAXxSgigBUw9v/jAR8239W8n+TRGQdA5CNGICoLy7U67CxqBwFP5biZFWD+Xh0oCdmpUfi9tQIhKpccOD0lQgCcPGs5WyzqiMArvB/SQpVp2AU0R6MIi8FJp9wwE0+FE9ARHbGoQJQfn4+Xn/9dajVaowbNw7Lli3D5MmTr3jdrl27cM011yAxMRHFxcXm42vXrsWDDz7Y5fzm5mYolb37JcQARP0hCAKKSuvwyY+l+PcBNRp0bQBMA6evGT0Ms9IjMWVsiOsOnO6N5otA1VFAW27a6FVTDmjKTC9tmenzK5IAPqGdwlF7C1Ln1iTPQK5hROSE+vL7W9RNkAoKCjB//nzk5+cjJycHb7/9NqZNm4bDhw8jKiqq2+s0Gg3uu+8+TJkyBefPd21C9/X1xbFjxyyO9Tb8EPWXRCJBapQ/UqP8sfjmBGw+WImPfyzF3jO1+PrYBXx97AICvOSYkTwcsydGYkyoiw+ctsbDH4jO6v5zXUN7OCrrFIw6wlKZKTAZdEC92vTCPuv3cVN2CkcRVlqThgNyr0F5RCKyD6K2AGVkZCA1NRUrV640Hxs7dixmzJiBvLy8bq+76667MGrUKMhkMmzcuLFLC9D8+fNRV1fX6zp0Oh10uksDWrVaLSIjI9kCRAPiTHUjPvmxFJ9eNnA6KUKFWRMjcUtSOHw5cHpgCALQWG0KRJcHpY5Xw3lcsZsNMIWxy8cfdX55hwIyJ9pIl8gJOEQLkF6vx/79+/HMM89YHM/NzcXu3bu7ve7dd9/FqVOn8P777+Oll16yek5DQwOio6NhMBiQnJyMF198ESkpKd3eMy8vD0uWLOnfgxBdQWyQF34/NR4LfjUaO09Uo2BfKbYfOY8DZRocKNPgxf8cxrTESwOnpVJ2zfSbRGJafNF7GDA81fo5bXqgvsJ6ONKWA3WlgL7e1N3WfNE0cNvqd8kA33Dr4agjOHn4DdqjEpFtRAtA1dXVMBgMCAkJsTgeEhKCyspKq9ecOHECzzzzDHbu3Ak3N+ulx8fHY+3atRg/fjy0Wi2WL1+OnJwcHDhwAKNGjbJ6zcKFC7FgwQLz+44WIKKB5CaT4rr4YFwXH4yaBh02FJlWnD5+vgEbisqxoagcUQGeuDMtAnekRyBMxfVxBoWbHPCPMb2606LpNP7ISmuSthwwtrV3vZUCpd3cx2sYEDTatG9a0OhLP6siAalsEB6OiHpL9Pbby6cJC4JgdeqwwWDAPffcgyVLlmD06NHd3i8zMxOZmZnm9zk5OUhNTcWbb76JFStWWL1GoVBAoVD08wmI+i7QW4GHJ4/AQ1fF4kCZBh//WIp/F1egpLYJfyk8jr9uP47Jo0wDp29ICIbCjb8sh5RSZXqFJFj/3GgAGqq6jj/q+FlbbtpQtuN1bpfl9W5K0+aylwejwJEce0Q0REQLQEFBQZDJZF1ae6qqqrq0CgFAfX09fvzxRxQVFeGxxx4DABiNRgiCADc3N2zbtg3XX399l+ukUikmTpyIEydODM6DENlAIpEgOdIPyZF+WHxTArb8okbBvlL8cKYW3x6/gG+PX4C/pztmpAzHrPRIjA3jmDS7IJUBvmGmV0S69XN0DUDNSaD6BFB9vP11wnSsrQU4/4vpdTlVZKdg1CkgeYdw5hrRABJ9EHRaWhry8/PNxxISEnDrrbd2GQRtNBpx+PBhi2P5+fn473//i08//RSxsbHw8ur6X06CIGDSpEkYP3481qxZ06u6OA2exHa2uhGf7i/Dp/vLUKm9tOnohAgV7kyPxK+TwqHy4MBph2Q0AHUlXYNR9XGgqbr76xS+1oORfyzXPSJq5zDrABUUFGDOnDn4+9//jqysLKxatQr/+Mc/cOjQIURHR2PhwoUoLy/He++9Z/X6F154ocsssCVLliAzMxOjRo2CVqvFihUr8K9//Qu7du3CpEmTelUXAxDZC4NRwI4TF/DJj6UoPHwerQbT/1wVblJMSwzFrPRIZI4I5MBpZ9FU2x6GjlkGo4tnAcFo/RqJDAiItT7WyMN/SMsnEptDzAIDgNmzZ6OmpgZLly6FWq1GYmIiNm/ejOjoaACAWq1GSUlJn+5ZV1eHuXPnorKyEiqVCikpKdixY0evww+RPZFJJbhuTDCuG2MaOL2xuAIf7yvFsfP12FhcgY3FFYgM8MCdaZGYmTIckQGeYpdMtvAMAKIyTK/O2nRA7emuLUbVJwB9e1dbzUng2GX363YQdhQg5YKc5NpEXwnaHrEFiOyZIAj4uX3g9KbiCtS3rzgNADGBnsiKC0LOyEBkjghEkDcH9zs1QTAt+GgRitp/1pZ3fx0HYZOTcpguMHvFAESOollvwNZDphWnfzhTC4PR8n/O8aE+yIoLRHZcECbFBnDckCvR1Xc/CNug7/66ywdhB44yLRngO5wLP5LdYwCyEQMQOSJtSyv2nanF7lM12H2qBkfUWovPpRJg/HAVsuKCkB0XiPQYf3jK+QvN5RgNQN25bgZh13R/ndTNFIL8owG/6PY/Yy699w7mLDUSHQOQjRiAyBnUNurx/eka7D5Vjd0na3C6utHic3eZBCmR/sgeaWohSo7040atrq6xBqi5PBidMK1v1FOrEQC4eQB+UZcFpE5/clVsGgIMQDZiACJnpNY0Y09769Duk9Wo0LRYfO7hLkN6jD+y21uIEoerIOPsMgIAo9E01qjuHHDxXNc/teW44v5qStVlwSim0/sowJ0rn5PtGIBsxABEzk4QBJTUNpm7y/acqkZ1g+V/4fso3ZARG4ic9hai0SHeVldpJ0Kb3tRK1F1A6ml9ow7eIdZbjvyjAd8Ijj+iXmEAshEDELkaQRBw/HyDqbvsVA2+P12D+pY2i3MCveTmAdXZcYGIDvRkIKLe0TWYFn/sLiDp63u+XiIDVMOtjz3yj+Yq2WTGAGQjBiBydQajgEMVGuw+VYNdJ6ux72wtWlotF+ILVynNA6qzRwZy81bqH0EAmi+aFnu0FpDqSnox/khp6kbrrgWJC0K6DAYgGzEAEVnStxlRXFpnbiEqKrloXpW6Q2yQlykMxQUhc0QAArkGEQ0EoxFoqLTeclTXPv6ou1WyOyhUpiAUMMK01lFgHBAQZ/rZM4CtR06EAchGDEBEPWvWG/DjuVrzgOqD5RpctgQR4kN9zN1lk0YEwFfJNYhoELTpAW1Z9wGp8ULP1ytVl8KQORi1v5SqoXkGGjAMQDZiACLqG01zK/aeqcXuU9XYc6oGRystx3RIJcD4CD/ktLcQpUX7w0MuE6lacin6RlM32sWzpu1Eak4CNadMP2tKe77WM6hTMBph+TNXzLZLDEA2YgAisk11g659DSJTC9HZmiaLz+UyKVKi/EwtRCMDkRTBNYhIBK3NQO0ZUyiqPWUKRjWnTD83nO/5Wp9wK8EozrQxrRu7f8XCAGQjBiCigVVe17EGkWlRxkpt1zWIJsYGmLrLYgMwKtgbPuwyIzHp6i+FoZrTliGpubb76yRSQBVhCkXm7rSRpqDkF83p/IOMAchGDEBEg0cQBJytaTKHoT2na1Db2HWWT6ivEiODvRE3zMv0Z7A3Rg7zxjAfBaffk7iaatu70051bT3qaUq/1M20AGTncUYdP/tGAFIHbwU1tAGtTaaWtdamy3628qdXMJA0e0BLYACyEQMQ0dAxGgUcO19v7i47UKZBdYOu2/N9lG4Y2R6GOkLRyGBvRAZ4cuVqEpcgmAZdm8cZtQekmtOmwNTW3P21bkrAP7ZrMAocafs6R4IAtOk6BZDLw8gVgkrnn/WNVu7R/rOxtW91RWYAD23r/3NZwQBkIwYgInFpmlpx8kIDTlU14NSFBpysasDJCw0orW3qMtusg1wmRWyQl7nVKC7YFIxGBHlzwDWJz2gE6is6BaNO441qz/QcHuTeprFFgSNNIUki6UVQuSzYXGmrkgElMQ0Sd/dof3le9mf7z4GjgGueHtBvZgCyEQMQkX1qaTXgbE0jTlY14FRVI062h6PTFxqga7O+FoxEAgz382gPRqZQ1PFzgJd8iJ+AyApDm2lG2uXBqOakaQbbldY56gupu2UQ6U1QMf95+efdnO+mEG1tJQYgGzEAETkWo1FAeV2zKRh1tBi1txrVNXX/X9YBXnJzV1rHWKORwd4IV3lAyu40sgdt+vYp/O3B6OJZQCrrRVCx9pkHIHPuyQUMQDZiACJyHjUNuvZg1GgORaeqGlBe1/14DA93GUZ0DL7u1GoUHegJhRu704jsFQOQjRiAiJxfk74Np9tDUUer0akLDThT3dhlm48OMqkEUQGeiBvmjbhgL/MA7Lhgb650TWQHGIBsxABE5LraDEaU1DZdajFqbzU6XdWAel1bt9cF+ygsxheNCvbG2DBf+HOcEdGQYQCyEQMQEV1OEARU1eu6tBidrGrAeW330/bDVUokhKuQEO6LceG+SAjzRYS/B9cyIhoEDEA2YgAior7QtrSau9M6gtHx8/U4d9kWIB18lW7tgUhlCkXhvogb5g13mYMvhEckMgYgGzEAEdFA0La04qi6HocqNDhcocWhCi1OVNVbHWMkd5NiTIgPxnW0FIX7Ij7UF14Kbp1A1FsMQDZiACKiwaJvM+JEVT0OVWhxuOOl1qLByvgiiQSIDfRCQnsgGheuQkKYL4b5cLNNImsYgGzEAEREQ8loFFB6scncSnSoQoPDam23Y4uCfRTmVqKOUBQV4Mm1i8jlMQDZiAGIiOxBdYOuSyg6U90Ia/+v7a1ww9gwH1Mgah9sPTrEB3I3jisi18EAZCMGICKyV426NhytrMfh9kB0qEKLo5X10FvZCsRdJsHIYB/z7LNx4b4YG+7LNYvIaTEA2YgBiIgcSavBiNMXGi0GWx+q0EDbYn3doqgAz0uhaLgvEsJUCPFVcGo+OTwGIBsxABGRoxME0/5oHYOtD1VocUSt7XYLkEAveZfB1rFBXpBxXBE5EAYgGzEAEZGzutioxxG15biik1UNMFr5TeDhLsOYUB+MDPZGbJAXRgR5IXaYF2ICvaB0555oZH8YgGzEAERErqSl1YBjle1T89Ua07gidT2aWw1Wz5dIgHCVB0YM8+oUjLwxIsgL4X4ebDUi0TAA2YgBiIhcncEo4Ex1I45WanHmQiNOV7e/LjSgvpuxRYBpQceYQE/EBnkhNsgUijqCUoCXnOOMaFD15fc3lxglIqIuZFKJeXPXzgRBQG2jHqerG83B6Ex1A85UN+JsdRP0bUYcP9+A4+cbAJy3uNZX6WZuKeroTjMFJS94yvnriIYWW4CsYAsQEVHfGYwCKuqa28NRQ3s4asTpC42o0DRbXb+oQ5hKaQ5DsUFeiBtmGncU4e8BN+6RRr3ELjAbMQAREQ2sllYDztZ0bjUydaedqW7ExabWbq9zk0oQFeiJEUHe5q60jnFHw3w4dZ8ssQuMiIjsitJdhvhQ0wavl7vYqMcZczhqMLcana1pREuraY2j0xcagSOW13kr3CxajUYM88KIIG/EBHnCh4s90hWI3gKUn5+P119/HWq1GuPGjcOyZcswefLkK163a9cuXHPNNUhMTERxcbHFZ+vXr8fixYtx6tQpxMXF4eWXX8bMmTN7XRNbgIiIxGc0CqjUtuD0BdM4o85damUXm6xO3e8wzEfR3pVmCkcxgV4Y7u+BcJUH/Dzd2XLkpBymC6ygoABz5sxBfn4+cnJy8Pbbb+Odd97B4cOHERUV1e11Go0GqampGDlyJM6fP28RgPbs2YPJkyfjxRdfxMyZM7FhwwY8//zz+O6775CRkdGruhiAiIjsm67NgNLaJlPrUPuA7DPtM9WqG6xvItvBw12GMD8lwlUeCFMpEebngfDL/vRWsIPEETlMAMrIyEBqaipWrlxpPjZ27FjMmDEDeXl53V531113YdSoUZDJZNi4caNFAJo9eza0Wi22bNliPjZ16lT4+/tj3bp1vaqLAYiIyHFpmltxtvpSIDLNUGtERV0zahr1vbqHj9LNFJD8lAhTdQ1IYSolF4O0Qw4xBkiv12P//v145plnLI7n5uZi9+7d3V737rvv4tSpU3j//ffx0ksvdfl8z549ePLJJy2O3XjjjVi2bFm399TpdNDpLv0Xg1ar7eVTEBGRvVF5uCMp0g9JkX5dPmtpNaBS04IKTTPUdS1Qa5pRoWlBRZ3pfYWmGfUtbahvacOxlnocO1/f7fcEeMlNLUgqD4T7XfozvD0ghfgq4c4ZbHZLtABUXV0Ng8GAkJAQi+MhISGorKy0es2JEyfwzDPPYOfOnXBzs156ZWVln+4JAHl5eViyZEkfn4CIiByN0l2GmCAvxAR5dXtOg64N6jpTMOr8p7pTcGpuNaC2UY/aRj0OVVj/j2aJBAj2UVgEpDDVpYAU7ueBYd4KSLlytihE7+S8fCCaIAhWB6cZDAbcc889WLJkCUaPHj0g9+ywcOFCLFiwwPxeq9UiMjKyN+UTEZGT8Va4YVSID0aF+Fj9XBAEaJpbUdGpBakjIJXXNUOtaUalpgWtBgHntTqc1+pQXGr9u9ykEoT4Ki8FpE5jkzqCElfQHhyiBaCgoCDIZLIuLTNVVVVdWnAAoL6+Hj/++COKiorw2GOPAQCMRiMEQYCbmxu2bduG66+/HqGhob2+ZweFQgGFQjEAT0VERM5OIpHAz1MOP085EsKtjzMxGgVUN+oudbNZCUvntS1oMwoor2tGeV0zgItW76Vwk3bqavNAhL8HIgM8Edn+Z4ivkvuv9YNoAUgulyMtLQ2FhYUWU9QLCwtx6623djnf19cXBw8etDiWn5+P//73v/j0008RGxsLAMjKykJhYaHFOKBt27YhOzt7kJ6EiIjIklQqQbCPEsE+SqtjkQCgzWBEVb3OMiDVtY9H0pjeVzfooWsz4mxNE87WNFm9j7tMguF+7aEowBOR/p6IDPBo/9MT/pz2b5WoXWALFizAnDlzkJ6ejqysLKxatQolJSWYN28eAFPXVHl5Od577z1IpVIkJiZaXB8cHAylUmlx/IknnsDVV1+NV199Fbfeeis+//xzbN++Hd99992QPhsREVFP3GRShPuZWnXSoq2fo2trH7RtDkjNKLvYjNKLTSitNb1vNQg9BiQvuQyRAZ6I8PdEVIBlOIoM8HDZfdhEferZs2ejpqYGS5cuhVqtRmJiIjZv3ozoaNO/CWq1GiUlJX26Z3Z2Nj766CM899xzWLx4MeLi4lBQUNDrNYCIiIjshcJNhuhAL0QHWh+03WYwolLbgpLaJpTVdgSjJpRebEZpbROq6nVo1BtwtLIeRyutz2gL9JIjor1LLeqyVqRwPw+nnckm+krQ9ojrABERkTNoaTV0ajHqeF16r21p6/F6qQQIU10ad3R5C5K9zWJzmIUQ7RUDEBERuQJNcytKa5tQ1t6lVlLbZA5HZReboWsz9ni93E1qCkftLUZR5tYj058qz6Hdk80hFkIkIiIicak83KEarkLicFWXz4xGAdUNOvN4o9Lapk4ByTTdX9/WabNaK3yUbhaDsqMCL3WvRfh7irqaNluArGALEBERUc9aDUao61o6jTu61IpUdrEJ1Q09bzsyMtgb2xdcM6A1sQWIiIiIBpW7TIqoQFOrjjVN+jbT+KNay4HZpRebUVbbhEh/jyGu2BIDEBEREQ04T7kbRof4YLSVFbUFQbji+KLB5pxz24iIiMhuSSQSUcf/AAxARERE5IIYgIiIiMjlMAARERGRy2EAIiIiIpfDAEREREQuhwGIiIiIXA4DEBEREbkcBiAiIiJyOQxARERE5HIYgIiIiMjlMAARERGRy2EAIiIiIpfDAEREREQux03sAuyRIAgAAK1WK3IlRERE1Fsdv7c7fo/3hAHIivr6egBAZGSkyJUQERFRX9XX10OlUvV4jkToTUxyMUajERUVFfDx8YFEIhnQe2u1WkRGRqK0tBS+vr4Dem/qO/592Bf+fdgX/n3YH/6d9EwQBNTX1yM8PBxSac+jfNgCZIVUKkVERMSgfoevry//5bUj/PuwL/z7sC/8+7A//Dvp3pVafjpwEDQRERG5HAYgIiIicjkMQENMoVDgj3/8IxQKhdilEPj3YW/492Ff+Pdhf/h3MnA4CJqIiIhcDluAiIiIyOUwABEREZHLYQAiIiIil8MARERERC6HAWgI5efnIzY2FkqlEmlpadi5c6fYJbmsvLw8TJw4ET4+PggODsaMGTNw7NgxscsimP5uJBIJ5s+fL3YpLq28vBy/+c1vEBgYCE9PTyQnJ2P//v1il+WS2tra8NxzzyE2NhYeHh4YMWIEli5dCqPRKHZpDo0BaIgUFBRg/vz5WLRoEYqKijB58mRMmzYNJSUlYpfmkr799ls8+uij+P7771FYWIi2tjbk5uaisbFR7NJc2r59+7Bq1SpMmDBB7FJc2sWLF5GTkwN3d3ds2bIFhw8fxl/+8hf4+fmJXZpLevXVV/H3v/8db731Fo4cOYLXXnsNr7/+Ot58802xS3NonAY/RDIyMpCamoqVK1eaj40dOxYzZsxAXl6eiJURAFy4cAHBwcH49ttvcfXVV4tdjktqaGhAamoq8vPz8dJLLyE5ORnLli0TuyyX9Mwzz2DXrl1spbYTN998M0JCQrB69Wrzsdtvvx2enp7417/+JWJljo0tQENAr9dj//79yM3NtTiem5uL3bt3i1QVdabRaAAAAQEBIlfiuh599FHcdNNNuOGGG8QuxeVt2rQJ6enpuPPOOxEcHIyUlBT84x//ELssl3XVVVfhq6++wvHjxwEABw4cwHfffYfp06eLXJlj42aoQ6C6uhoGgwEhISEWx0NCQlBZWSlSVdRBEAQsWLAAV111FRITE8UuxyV99NFH+Omnn7Bv3z6xSyEAp0+fxsqVK7FgwQI8++yz2Lt3L/7v//4PCoUC9913n9jluZw//OEP0Gg0iI+Ph0wmg8FgwMsvv4y7775b7NIcGgPQEJJIJBbvBUHocoyG3mOPPYaff/4Z3333ndiluKTS0lI88cQT2LZtG5RKpdjlEACj0Yj09HS88sorAICUlBQcOnQIK1euZAASQUFBAd5//318+OGHGDduHIqLizF//nyEh4fj/vvvF7s8h8UANASCgoIgk8m6tPZUVVV1aRWiofX4449j06ZN2LFjByIiIsQuxyXt378fVVVVSEtLMx8zGAzYsWMH3nrrLeh0OshkMhErdD1hYWFISEiwODZ27FisX79epIpc29NPP41nnnkGd911FwBg/PjxOHfuHPLy8hiAbMAxQENALpcjLS0NhYWFFscLCwuRnZ0tUlWuTRAEPPbYY/jss8/w3//+F7GxsWKX5LKmTJmCgwcPori42PxKT0/Hvffei+LiYoYfEeTk5HRZFuL48eOIjo4WqSLX1tTUBKnU8te1TCbjNHgbsQVoiCxYsABz5sxBeno6srKysGrVKpSUlGDevHlil+aSHn30UXz44Yf4/PPP4ePjY26dU6lU8PDwELk61+Lj49Nl7JWXlxcCAwM5JkskTz75JLKzs/HKK69g1qxZ2Lt3L1atWoVVq1aJXZpLuuWWW/Dyyy8jKioK48aNQ1FREd544w389re/Fbs0h8Zp8EMoPz8fr732GtRqNRITE/HXv/6VU65F0t3Yq3fffRcPPPDA0BZDXVx77bWcBi+y//znP1i4cCFOnDiB2NhYLFiwAI888ojYZbmk+vp6LF68GBs2bEBVVRXCw8Nx99134/nnn4dcLhe7PIfFAEREREQuh2OAiIiIyOUwABEREZHLYQAiIiIil8MARERERC6HAYiIiIhcDgMQERERuRwGICIiInI5DEBERETkchiAiIh6QSKRYOPGjWKXQUQDhAGIiOzeAw88AIlE0uU1depUsUsjIgfFzVCJyCFMnToV7777rsUxhUIhUjVE5OjYAkREDkGhUCA0NNTi5e/vD8DUPbVy5UpMmzYNHh4eiI2NxSeffGJx/cGDB3H99dfDw8MDgYGBmDt3LhoaGizOWbNmDcaNGweFQoGwsDA89thjFp9XV1dj5syZ8PT0xKhRo7Bp06bBfWgiGjQMQETkFBYvXozbb78dBw4cwG9+8xvcfffdOHLkCACgqakJU6dOhb+/P/bt24dPPvkE27dvtwg4K1euxKOPPoq5c+fi4MGD2LRpE0aOHGnxHUuWLMGsWbPw888/Y/r06bj33ntRW1s7pM9JRANEICKyc/fff78gk8kELy8vi9fSpUsFQRAEAMK8efMsrsnIyBD+93//VxAEQVi1apXg7+8vNDQ0mD//4osvBKlUKlRWVgqCIAjh4eHCokWLuq0BgPDcc8+Z3zc0NAgSiUTYsmXLgD0nEQ0djgEiIodw3XXXYeXKlRbHAgICzD9nZWVZfJaVlYXi4mIAwJEjR5CUlAQvLy/z5zk5OTAajTh27BgkEgkqKiowZcqUHmuYMGGC+WcvLy/4+Pigqqqqv49ERCJiACIih+Dl5dWlS+pKJBIJAEAQBPPP1s7x8PDo1f3c3d27XGs0GvtUExHZB44BIiKn8P3333d5Hx8fDwBISEhAcXExGhsbzZ/v2rULUqkUo0ePho+PD2JiYvDVV18Nac1EJB62ABGRQ9DpdKisrLQ45ubmhqCgIADAJ598gvT0dFx11VX44IMPsHfvXqxevRoAcO+99+KPf/wj7r//frzwwgu4cOECHn/8ccyZMwchISEAgBdeeAHz5s1DcHAwpk2bhvr6euzatQuPP/740D4oEQ0JBiAicghffvklwsLCLI6NGTMGR48eBWCaofXRRx/hd7/7HUJDQ/HBBx8gISEBAODp6YmtW7fiiSeewMSJE+Hp6Ynbb78db7zxhvle999/P1paWvDXv/4VTz31FIKCgnDHHXcM3QMS0ZCSCIIgiF0EEZEtJBIJNmzYgBkzZohdChE5CI4BIiIiIpfDAEREREQuh2OAiMjhsSefiPqKLUBERETkchiAiIiIyOUwABEREZHLYQAiIiIil8MARERERC6HAYiIiIhcDgMQERERuRwGICIiInI5/x9FAHKFrKtfrQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.plot(trainer.get_hist_metric('fit'), label='fit')\n",
    "plt.plot(trainer.get_hist_metric('test'), label='train')\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Train a SNN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Similarly, ``brainpy.train.BPTT`` can also be used to train SNN models.\n",
    "\n",
    "We first build a three layer SNN model:\n",
    "\n",
    "```bash\n",
    "\n",
    "i >> [exponential synapse] >> r >> [exponential synapse] >> o\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T08:51:17.878791500Z",
     "start_time": "2024-01-13T08:51:17.851882800Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class SNNModel(bp.DynamicalSystem):\n",
    "  def __init__(self, num_in, num_rec, num_out):\n",
    "    super(SNNModel, self).__init__()\n",
    "\n",
    "    # parameters\n",
    "    self.num_in = num_in\n",
    "    self.num_rec = num_rec\n",
    "    self.num_out = num_out\n",
    "\n",
    "    # neuron groups\n",
    "    self.r = bp.dyn.LifRef(num_rec, tau=10, V_reset=0, V_rest=0, V_th=1.)\n",
    "    self.o = bp.dyn.Leaky(num_out, tau=5)\n",
    "\n",
    "    # synapse: i->r\n",
    "    self.i2r = bp.dyn.HalfProjAlignPost(comm=bp.dnn.Linear(num_in, num_rec, bp.init.KaimingNormal(scale=2.)),\n",
    "                                        syn=bp.dyn.Expon(num_rec, tau=10.),\n",
    "                                        out=bp.dyn.CUBA(),\n",
    "                                        post=self.r)\n",
    "    # synapse: r->o\n",
    "    self.r2o = bp.dyn.HalfProjAlignPost(comm=bp.dnn.Linear(num_rec, num_out, bp.init.KaimingNormal(scale=2.)),\n",
    "                                        syn=bp.dyn.Expon(num_out, tau=10.),\n",
    "                                        out=bp.dyn.CUBA(),\n",
    "                                        post=self.o)\n",
    "\n",
    "  def update(self, spike):\n",
    "    self.i2r(spike)\n",
    "    self.r2o(self.r.spike.value)\n",
    "    self.r()\n",
    "    self.o()\n",
    "    return self.o.x.value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "As the model receives spiking inputs, we define functions that are necessary to transform the continuous values to spiking data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T08:50:19.098345900Z",
     "start_time": "2024-01-13T08:50:19.091227600Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def current2firing_time(x, tau=20., thr=0.2, tmax=1.0, epsilon=1e-7):\n",
    "  x = np.clip(x, thr + epsilon, 1e9)\n",
    "  T = tau * np.log(x / (x - thr))\n",
    "  T = np.where(x < thr, tmax, T)\n",
    "  return T\n",
    "\n",
    "def sparse_data_generator(X, y, batch_size, nb_steps, nb_units, shuffle=True):\n",
    "  labels_ = np.array(y, dtype=bm.int_)\n",
    "  sample_index = np.arange(len(X))\n",
    "\n",
    "  # compute discrete firing times\n",
    "  tau_eff = 2. / bm.get_dt()\n",
    "  unit_numbers = np.arange(nb_units)\n",
    "  firing_times = np.array(current2firing_time(X, tau=tau_eff, tmax=nb_steps), dtype=bm.int_)\n",
    "\n",
    "  if shuffle:\n",
    "    np.random.shuffle(sample_index)\n",
    "\n",
    "  counter = 0\n",
    "  number_of_batches = len(X) // batch_size\n",
    "  while counter < number_of_batches:\n",
    "    batch_index = sample_index[batch_size * counter:batch_size * (counter + 1)]\n",
    "    all_batch, all_times, all_units = [], [], []\n",
    "    for bc, idx in enumerate(batch_index):\n",
    "      c = firing_times[idx] < nb_steps\n",
    "      times, units = firing_times[idx][c], unit_numbers[c]\n",
    "      batch = bc * np.ones(len(times), dtype=bm.int_)\n",
    "      all_batch.append(batch)\n",
    "      all_times.append(times)\n",
    "      all_units.append(units)\n",
    "    all_batch = np.concatenate(all_batch).flatten()\n",
    "    all_times = np.concatenate(all_times).flatten()\n",
    "    all_units = np.concatenate(all_units).flatten()\n",
    "    x_batch = bm.zeros((batch_size, nb_steps, nb_units))\n",
    "    x_batch[all_batch, all_times, all_units] = 1.\n",
    "    y_batch = bm.asarray(labels_[batch_index])\n",
    "    yield x_batch, y_batch\n",
    "    counter += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now, we can define a BP trainer for this SNN model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T08:51:22.363907900Z",
     "start_time": "2024-01-13T08:51:21.746626200Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def loss_fun(predicts, targets):\n",
    "    predicts, mon = predicts\n",
    "    # L1 loss on total number of spikes\n",
    "    l1_loss = 1e-5 * bm.sum(mon['r.spike'])\n",
    "    # L2 loss on spikes per neuron\n",
    "    l2_loss = 1e-5 * bm.mean(bm.sum(bm.sum(mon['r.spike'], axis=0), axis=0) ** 2)\n",
    "    # predictions\n",
    "    predicts = bm.max(predicts, axis=1)\n",
    "    loss = bp.losses.cross_entropy_loss(predicts, targets)\n",
    "    acc = bm.mean(predicts.argmax(-1) == targets)\n",
    "    return loss + l2_loss + l1_loss, {'acc': acc}\n",
    "\n",
    "model = SNNModel(num_in=28*28, num_rec=100, num_out=10)\n",
    "\n",
    "trainer = bp.BPTT(\n",
    "    model,\n",
    "    loss_fun=loss_fun,\n",
    "    loss_has_aux=True,\n",
    "    optimizer=bp.optim.Adam(lr=1e-3),\n",
    "    monitors={'r.spike': model.r.spike},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "The training process is similar to that of the ANN model, instead of the data is generated by the sparse generator function we defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:02:11.510933Z",
     "start_time": "2024-01-13T08:51:23.628031300Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train 0 epoch, use 81.7961 s, loss 1.7836289405822754, acc 0.26856303215026855\n",
      "Train 1 epoch, use 110.9031 s, loss 1.716126561164856, acc 0.28009817004203796\n",
      "Train 2 epoch, use 121.7257 s, loss 1.703003168106079, acc 0.28330329060554504\n",
      "Train 3 epoch, use 152.4789 s, loss 1.6957000494003296, acc 0.2849225401878357\n",
      "Train 4 epoch, use 180.2322 s, loss 1.6888805627822876, acc 0.2862913906574249\n"
     ]
    }
   ],
   "source": [
    "x_train = bm.array(train_dataset.data, dtype=bm.float_) / 255\n",
    "y_train = bm.array(train_dataset.targets, dtype=bm.int_)\n",
    "\n",
    "trainer.fit(lambda: sparse_data_generator(x_train.reshape(x_train.shape[0], -1),\n",
    "                                          y_train,\n",
    "                                          batch_size=256,\n",
    "                                          nb_steps=100,\n",
    "                                          nb_units=28 * 28),\n",
    "            num_epoch=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:05:36.267895100Z",
     "start_time": "2024-01-13T09:05:36.138927700Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABD50lEQVR4nO3deXRV9b3//9c5J3PIyJABMgAqo4QQJKAgUKmVelHEoVVAsFJFbeuwur5fqUNv7a/S9nbiXr5i0SDOthVB71UseksCCogMQRBEEzKRwQCZE8hwzv79ceC0MQGSnCT7DM/HWnstzz6fc/L++FHy4vP57L0thmEYAgAA8CNWswsAAADobwQgAADgdwhAAADA7xCAAACA3yEAAQAAv0MAAgAAfocABAAA/E6A2QV4IofDobKyMkVERMhisZhdDgAA6ALDMFRfX6/ExERZrRee4yEAdaKsrExJSUlmlwEAAHqgpKREw4YNu2AbAlAnIiIiJDn/BUZGRppcDQAA6Iq6ujolJSW5fo9fCAGoE+eWvSIjIwlAAAB4ma5sX2ETNAAA8DsEIAAA4HcIQAAAwO+wBwgAAC9lt9vV2tpqdhn9Kigo6KKXuHcFAQgAAC9jGIYqKipUU1Njdin9zmq1avjw4QoKCnLrewhAAAB4mXPhZ8iQIQoLC/Obm/aeu1FxeXm5kpOT3eo3AQgAAC9it9td4WfgwIFml9PvBg8erLKyMrW1tSkwMLDH38MmaAAAvMi5PT9hYWEmV2KOc0tfdrvdre8hAAEA4IX8Zdnrm3qr3wQgAADgdwhAAADA7xCAAABAvzAMQ/fcc49iY2NlsVgUHR2thx56yJRauAqsn5XXnlZNU6vGJPCQVQCAf3n//fe1fv16ZWdna8SIEbJarQoNDXW9n5qaqoceeqhfQhEBqB9tPliun7yxX+OHRumt+6702w1sAAD/lJ+fr4SEBF155ZVml0IA6k8ZqTGyWCzaX1yjTwqqNHWE/92/AQDQ+wzD0OlW9y4L74nQQFuX/zK/dOlSvfjii5KcV3KlpKQoNTVVEydO1J/+9CfNmjVLRUVFevjhh/Xwww9LcvarrxCA+tGQiBDdNnmYXtlVrDXZ+QQgAECvON1q19gn/97vP/fwU99RWFDXosSqVas0cuRIrV27Vp9++qlsNptuvfVW1/tvvfWW0tLSdM899+iHP/xhX5XswibofnbPjJGyWqScL0/oUGmt2eUAANAvoqKiFBERIZvNpvj4eA0ePLjd+7GxsbLZbIqIiFB8fLzi4+P7tB5mgPpZ8sAwzUtL1Nu5ZXo2J1+r75hkdkkAAC8XGmjT4ae+Y8rP9VYEIBMsnzlSb+eW6b2D5So82ajUQeFmlwQA8GIWi6XLS1FwYgnMBGMSIvWt0UPkMKQ/bztmdjkAAHiEoKAgt5/x1VUEIJPcP2ukJGnD3uP6uu6MydUAAGC+1NRUbdu2TaWlpTp58mSf/iwCkEkmp8bqitQYtdgdyvqowOxyAAAw3VNPPaXCwkKNHDmywybp3mYx+vIiey9VV1enqKgo1dbWKjKy7+7YvPWLSt21/lOFB9m049FrFBUW2Gc/CwDgG86cOaOCggINHz5cISEhZpfT7y7U/+78/jZ1Bmjbtm2aN2+eEhMTZbFYtGnTpgu2X7p0qSwWS4dj3Lhx7dr96U9/0qhRoxQaGqqkpCQ9/PDDOnPG85aZZo0arNHxEWpsseulnYVmlwMAgN8wNQA1NjYqLS1Nq1ev7lL7VatWqby83HWUlJQoNja23Y2UXn31VT366KP6+c9/riNHjigrK0t/+ctftGLFir7qRo9ZLBbdd3Yv0As7CnW6pf/v4gkAgD8y9Zq5uXPnau7cuV1uHxUVpaioKNfrTZs2qbq6WnfddZfr3M6dO3XVVVfpjjvukOTcUHX77bdr9+7dvVd4L7r+8gT9fsuXKq5q0l8+LdbSq4abXRIAAD7PqzdBZ2Vlac6cOUpJSXGdmz59uvbu3esKPMeOHdN7772n66+//rzf09zcrLq6unZHfwmwWXXP1SMkSc9tL1Cr3dFvPxsA4L38dQtvb/XbawNQeXm5Nm/erGXLlrU7//3vf1+//OUvNX36dAUGBmrkyJGaPXu2Hn300fN+18qVK12zS1FRUUpKSurr8tu5JWOYBg0IVmnNab2TW9avPxsA4F0CA50XzDQ1NZlciTlaWlokSTabe3eh9trbRq5fv17R0dGaP39+u/PZ2dn61a9+pWeeeUaZmZnKy8vTgw8+qISEBD3xxBOdfteKFSv0yCOPuF7X1dX1awgKCbTp7unD9Zv3v9CzOfm6KX2orNauPV0XAOBfbDaboqOjVVlZKUkKCwvr8hPZvZ3D4dCJEycUFhamgAD3IoxXBiDDMLRu3TotXrxYQUFB7d574okntHjxYtfM0OWXX67Gxkbdc889euyxx2S1dpz0Cg4OVnBwcL/Ufj6Lpibrmew8fVXZoA+PfK1rx/XtQ+AAAN7r3INCz4Ugf2K1WpWcnOx26PPKAJSTk6O8vDzdfffdHd5ramrqEHJsNpsMw/Do9dKIkEAtnpqiZ7Lz9Ux2vr49Ns5vEj0AoHssFosSEhI0ZMgQtba2ml1OvwoKCup0MqO7TA1ADQ0NysvLc70uKChQbm6uYmNjlZycrBUrVqi0tFQvvfRSu89lZWUpMzNT48eP7/Cd8+bN0x/+8Aelp6e7lsCeeOIJ3XDDDW6vF/a1u64arqyPCpRbUqNdx6o0beRAs0sCAHgwm83m8b/bPJWpAWjPnj2aPXu26/W5fThLlizR+vXrVV5eruLi4nafqa2t1YYNG7Rq1apOv/Pxxx+XxWLR448/rtLSUg0ePFjz5s3Tr371q77rSC8ZHBGs2yYn6eVdRXomO48ABABAH+FRGJ3or0dhdKakqkmzfpctu8PQ//x4usYPjbr4hwAAgPc8CgMdJcWGad6EBEnSmux8k6sBAMA3EYA80PKzj8d471C5Ck42mlwNAAC+hwDkgUbHR+qa0UNkGNKfc5gFAgCgtxGAPNT9s52zQBv2HVdFrec9yR4AAG9GAPJQGSmxmpIaq1a7oayPjpldDgAAPoUA5MHuOzsL9OonxappajG5GgAAfAcByIPNumywxiREqqnFrpd2FpldDgAAPoMA5MEsFovuO3tF2AsfF6ippc3kigAA8A0EIA/33fHxShkYpuqmVv3l0xKzywEAwCcQgDxcgM2qe64eIUl6btsxtbQ5TK4IAADvRwDyAjdPGqbBEcEqqz2jdw6UmV0OAABejwDkBUICbbp7+nBJ0rM5+XI4eHwbAADuIAB5iYWZyYoICVBeZYM+OPK12eUAAODVCEBeIiIkUHdOS5EkPZOdL8NgFggAgJ4iAHmRu64aruAAqw6U1Ghn/imzywEAwGsRgLzIoAHB+v4VSZKkNTwkFQCAHiMAeZllM0bIZrVo+1cndfB4rdnlAADglQhAXiYpNkw3piVKktbk5JlcDQAA3okA5IWWn308xuZDFco/0WByNQAAeB8CkBe6LC5Cc8bEyTCktTnHzC4HAACvQwDyUucekvrW/uMqrz1tcjUAAHgXApCXykiJUebwWLXaDWVtLzC7HAAAvAoByIudmwV6bXexqhtbTK4GAADvQQDyYjMvG6xxiZFqarHrxZ2FZpcDAIDXIAB5MYvF4poFWr+jUE0tbSZXBACAdyAAebm54xOUOjBMNU2ten13idnlAADgFQhAXs5mtejemc5ZoOe3H1NLm8PkigAA8HwEIB+wYNJQDYkIVnntGW3KLTW7HAAAPB4ByAcEB9i0bMZwSdKzOflyOAyTKwIAwLMRgHzEHZkpigwJ0LETjdpyuMLscgAA8GgEIB8xIDhAS65MlSStyc6XYTALBADA+RCAfMjSK1MVEmjVgeO12pF/yuxyAADwWAQgHzJwQLC+f0WyJOmZ7DyTqwEAwHMRgHzMshnDFWC16OO8UzpQUmN2OQAAeCQCkI8ZFhOmGyYmSnLuBQIAAB0RgHzQfWdvjPj3wxXKq2wwuRoAADwPAcgHXRoXoW+PjZNhSH/OYRYIAIBvIgD5qHMPSd2UW6qymtMmVwMAgGchAPmoSckxmjoiVq12Q89vLzC7HAAAPAoByIfdP+sSSdLru4tV3dhicjUAAHgOApAPm3HpII1LjNTpVrvW7yg0uxwAADwGAciHWSwW1yzQizsL1djcZnJFAAB4BgKQj7tufLyGDwpXTVOrXt9dbHY5AAB4BAKQj7NZLbr36hGSpOe3F6i5zW5yRQAAmI8A5AdumjRUcZHBqqg7o7f3l5ldDgAApiMA+YHgAJuWTXfOAj2bky+7wzC5IgAAzEUA8hO3ZyYrKjRQx042asvnFWaXAwCAqQhAfmJAcICWTEuRJD2TnS/DYBYIAOC/CEB+ZOlVwxUSaNXB0lp9nHfK7HIAADANAciPxIYH6ftXJEuSnsnOM7kaAADMQwDyMz+8eoQCrBbtyD+l3JIas8sBAMAUBCA/MzQ6VPPTh0qS1jALBADwUwQgP7R85ghZLNLfP/9aeZX1ZpcDAEC/IwD5oUuGROjasXGSpGdzjplcDQAA/Y8A5KfuO/uQ1E37S1Vac9rkagAA6F8EID81MSlaV44cqDaHoee3MwsEAPAvBCA/dt+skZKkN3aXqKqxxeRqAADoPwQgPzb9kkG6fGiUTrfatX5HodnlAADQbwhAfsxisbhmgV7cUaiG5jaTKwIAoH8QgPzcd8bFa8SgcNWebtUbu4vNLgcAgH5hagDatm2b5s2bp8TERFksFm3atOmC7ZcuXSqLxdLhGDduXLt2NTU1euCBB5SQkKCQkBCNGTNG7733Xh/2xHvZrBbdO3OEJOm57cfU3GY3uSIAAPqeqQGosbFRaWlpWr16dZfar1q1SuXl5a6jpKREsbGxuvXWW11tWlpa9O1vf1uFhYV68803dfToUT333HMaOnRoX3XD692UPkzxkSH6uq5ZG/eVml0OAAB9LsDMHz537lzNnTu3y+2joqIUFRXler1p0yZVV1frrrvucp1bt26dqqqqtGPHDgUGBkqSUlJSeq9oHxQUYNWyGcP1/717RH/edky3Tk6SzWoxuywAAPqMV+8BysrK0pw5c9oFnHfeeUfTpk3TAw88oLi4OI0fP15PP/207PbzL+00Nzerrq6u3eFvbp+SrOiwQBWcbNT7hyrMLgcAgD7ltQGovLxcmzdv1rJly9qdP3bsmN58803Z7Xa99957evzxx/X73/9ev/rVr877XStXrnTNLkVFRSkpKamvy/c44cEBWjItVZK0JidPhmGYWxAAAH3IawPQ+vXrFR0drfnz57c773A4NGTIEK1du1YZGRn6/ve/r8cee0xr1qw573etWLFCtbW1rqOkpKSPq/dMS69MVWigTYdK67T9q5NmlwMAQJ/xygBkGIbWrVunxYsXKygoqN17CQkJuuyyy2Sz2VznxowZo4qKCrW0dH634+DgYEVGRrY7/FFMeJBun5IsSVqTnW9yNQAA9B2vDEA5OTnKy8vT3Xff3eG9q666Snl5eXI4HK5zX375pRISEjqEJXS0bMZwBdos2nnslPYXV5tdDgAAfcLUANTQ0KDc3Fzl5uZKkgoKCpSbm6viYucN+VasWKE777yzw+eysrKUmZmp8ePHd3jvvvvu06lTp/Tggw/qyy+/1Lvvvqunn35aDzzwQJ/2xVckRodq/kTnLQOYBQIA+CpTA9CePXuUnp6u9PR0SdIjjzyi9PR0Pfnkk5KcG53PhaFzamtrtWHDhk5nfyQpKSlJW7Zs0aeffqoJEyboJz/5iR588EE9+uijfdsZH3LvzJGyWKQth7/WV1/Xm10OAAC9zmJwuU8HdXV1ioqKUm1trd/uB1r+8l69/3mFFkwaqj/cNtHscgAAuKju/P72yj1A6HvnHpL6Tm6Zjlc3mVwNAAC9iwCETqUlReuqSwaqzWHo+e0FZpcDAECvIgDhvO6fdYkk6Y1Pi3WqodnkagAA6D0EIJzXlSMHasKwKJ1pdWj9jkKzywEAoNcQgHBeFotF95/dC/TijkI1NLeZXBEAAL2DAIQLunZsvEYMDlfdmTa99kmR2eUAANArCEC4IKvVouUznbNAz28vUHOb3eSKAABwHwEIFzV/4lAlRIWosr5Zb+0rNbscAADcRgDCRQUFWLVsxghJ0p9z8mV3cO9MAIB3IwChS26fkqTosEAVnmrS5kPlZpcDAIBbCEDokrCgAC29MlWS9MzWfPEEFQCANyMAocuWTEtVWJBNh8vrtO2rk2aXAwBAjxGA0GUx4UG6fUqyJOmZrXkmVwMAQM8RgNAty2YMV6DNok8KqrS3qNrscgAA6BECELolISpUN6UPlSStyc43uRoAAHqGAIRuu3fmSFks0odHvtaXX9ebXQ4AAN1GAEK3jRw8QNeNi5ckPcssEADACxGA0CP3z7pEkvT2gTKVVDWZXA0AAN1DAEKPXD4sSjMuHSS7w9Dz24+ZXQ4AAN1CAEKP3Xf2IalvfFqikw3NJlcDAEDXEYDQY9NGDlRaUrSa2xxa/3Gh2eUAANBlBCD0mMVicc0CvbizUPVnWk2uCACAriEAwS3Xjo3TyMHhqj/Tptc+KTa7HAAAuoQABLdYrRYtPzsL9PxHBTrTaje5IgAALo4ABLfdOHGoEqNCdKK+WW/tKzW7HAAALooABLcFBVi1bMYISdKft+Wrze4wuSIAAC6MAIRe8f0pSYoJC1TRqSa9d6jC7HIAALggAhB6RVhQgO66argk50NSDcMwuSIAAM6PAIRec+e0FIUH2XSkvE7ZX54wuxwAAM6LAIReEx0WpDsykyU5Z4EAAPBUBCD0qrunj1CgzaLdBVXaW1RldjkAAHSKAIReFR8VopsnDZPELBAAwHMRgNDr7rl6hCwW6cMjlTpaUW92OQAAdEAAQq8bMXiAvjs+QZL0bA6zQAAAz0MAQp+4b5bz8RjvHChTSVWTydUAANAeAQh9YvzQKM24dJDsDkNrtx0zuxwAANohAKHP3D/rEknSX/eU6ER9s8nVAADwTwQg9JmpI2I1MSlazW0OvfBxgdnlAADgQgBCn7FYLLr/7F6gl3cWqe5Mq8kVAQDgRABCn5ozJk6XDhmg+uY2vbqr2OxyAACQRABCH7NaLVo+0zkLlPVRgc602k2uCAAAAhD6wQ0TEzU0OlQnG5r15t7jZpcDAAABCH0v0GbVD2cMlySt3XZMbXaHyRUBAPwdAQj94ntXJCs2PEjFVU1692C52eUAAPwcAQj9IjTIpruuTJXkfEiqYRjmFgQA8GsEIPSbO6elKjzIpi8q6pV99ITZ5QAA/BgBCP0mKixQC6emSJKeyc4zuRoAgD8jAKFf3T19uIJsVn1aWK1PC6vMLgcA4KcIQOhXcZEhujljqCTnXiAAAMxAAEK/u/fqkbJapH98Uakj5XVmlwMA8EMEIPS71EHhmnt5giTp2RxmgQAA/Y8ABFPcd/bxGP99oEzFp5pMrgYA4G8IQDDF+KFRuvqywXIY0trtzAIBAPoXAQimuX+Wcxbor3uOq7L+jMnVAAD8CQEIpskcHqv05Gi1tDn0wseFZpcDAPAjBCCYxmKx6P5Zl0iSXtlZpLozrSZXBADwFwQgmOqa0UN0WdwA1Te36ZVdRWaXAwDwEwQgmMpqtWj52SvC1n1UoDOtdpMrAgD4AwIQTDcvLVFDo0N1sqFFf9t73OxyAAB+gAAE0wXarLrn6hGSpLXb8tVmd5hcEQDA15kagLZt26Z58+YpMTFRFotFmzZtumD7pUuXymKxdDjGjRvXafs33nhDFotF8+fP7/3i0atum5ykgeFBKqk6rXcPlptdDgDAx5kagBobG5WWlqbVq1d3qf2qVatUXl7uOkpKShQbG6tbb721Q9uioiL99Kc/1YwZM3q7bPSB0CCb7roqVZLzIamGYZhbEADApwWY+cPnzp2ruXPndrl9VFSUoqKiXK83bdqk6upq3XXXXe3a2e12LVy4UL/4xS+0fft21dTU9FbJ6EOLp6Xq2Zxj+qKiXv/4olLXjIkzuyQAgI/y6j1AWVlZmjNnjlJSUtqdf+qppzR48GDdfffdXfqe5uZm1dXVtTvQ/6JCA7UwM1mScxYIAIC+4rUBqLy8XJs3b9ayZcvanf/444+VlZWl5557rsvftXLlStfsUlRUlJKSknq7XHTR3dOHKyjAqj1F1dpdUGV2OQAAH+W1AWj9+vWKjo5ut8G5vr5eixYt0nPPPadBgwZ1+btWrFih2tpa11FSUtIHFaMrhkSG6JaMYZKkNdl5JlcDAPBVPdoDVFJSIovFomHDnL+odu/erddee01jx47VPffc06sFdsYwDK1bt06LFy9WUFCQ63x+fr4KCws1b9481zmHw3lJdUBAgI4ePaqRI0d2+L7g4GAFBwf3ed3omnuvHqE3dhdr69ETOlxWp7GJkWaXBADwMT2aAbrjjju0detWSVJFRYW+/e1va/fu3frZz36mp556qlcL7ExOTo7y8vI67PEZPXq0Dh48qNzcXNdxww03aPbs2crNzWVpy0ukDAzX9RMSJUnP5rAXCADQ+3oUgA4dOqQpU6ZIkv76179q/Pjx2rFjh1577TWtX7++y9/T0NDgCiqSVFBQoNzcXBUXF0tyLk3deeedHT6XlZWlzMxMjR8/vt35kJAQjR8/vt0RHR2tiIgIjR8/vt1sETzb8pnOGyP+z2dlKjrVaHI1AABf06MA1Nra6loy+vDDD3XDDTdIcs7AlJd3/SZ2e/bsUXp6utLT0yVJjzzyiNLT0/Xkk09Kcm50PheGzqmtrdWGDRu6fIUXvNO4xCjNGjVYDkNau+2Y2eUAAHyMxejBHecyMzM1e/ZsXX/99br22mu1a9cupaWladeuXbrlllt0/Lh3P8+prq5OUVFRqq2tVWQk+0/M8smxU/re2l0KCrDqo/87W0MiQswuCQDgwbrz+7tHM0C/+c1v9Oc//1mzZs3S7bffrrS0NEnSO++841oaA9w1ZXisMlJi1NLmUNZHBWaXAwDwIT2aAZKcd1uuq6tTTEyM61xhYaHCwsI0ZMiQXivQDMwAeY4PD3+tZS/t0YDgAH386LcUFRpodkkAAA/V5zNAp0+fVnNzsyv8FBUV6U9/+pOOHj3q9eEHnuVbo4doVFyEGprb9MquIrPLAQD4iB4FoBtvvFEvvfSSJKmmpkaZmZn6/e9/r/nz52vNmjW9WiD8m9Vq0X2znPduWvdRgc602k2uCADgC3oUgPbt2+d6yvqbb76puLg4FRUV6aWXXtJ//ud/9mqBwL9NSNCwmFCdamzRX/dwl24AgPt6FICampoUEREhSdqyZYsWLFggq9WqqVOnqqiIZQr0rgCbVfde7bwv0J9zjqnV7jC5IgCAt+tRALrkkku0adMmlZSU6O9//7uuvfZaSVJlZSWbhtEnbp2cpEEDglRac1r/81mZ2eUAALxcjwLQk08+qZ/+9KdKTU3VlClTNG3aNEnO2aBzNzUEelNIoE13XTVckrQmO18OR48uXgQAQJIbl8FXVFSovLxcaWlpslqdOWr37t2KjIzU6NGje7XI/sZl8J6p9nSrrvr1P9TQ3Kbn75ysOWPjzC4JAOBB+vwyeEmKj49Xenq6ysrKVFpaKkmaMmWK14cfeK6o0EAtmpoiSXomO089zO4AAPQsADkcDj311FOKiopSSkqKkpOTFR0drV/+8pdyONigir7zg+mpCgqwal9xjXYXVJldDgDAS/UoAD322GNavXq1fv3rX2v//v3at2+fnn76af3Xf/2Xnnjiid6uEXAZEhGiWzOGSZKeyc43uRoAgLfq0R6gxMREPfvss66nwJ/z9ttv6/7773ctiXkr9gB5tuJTTZr1u61yGNK7P5mucYlRZpcEAPAAfb4HqKqqqtO9PqNHj1ZVFcsS6FvJA8P0bxMSJTmvCAMAoLt6FIDS0tK0evXqDudXr16tCRMmuF0UcDHnHo/x3sFyFZ5sNLkaAIC3CejJh37729/q+uuv14cffqhp06bJYrFox44dKikp0XvvvdfbNQIdjEmI1OxRg7X16An9edsxrVxwudklAQC8SI9mgGbOnKkvv/xSN910k2pqalRVVaUFCxbo888/1wsvvNDbNQKdun/2JZKkDXuPq7LujMnVAAC8SY9vhNiZAwcOaNKkSbLbvfuJ3WyC9h63rNmhPUXVuvfqEVrx3TFmlwMAMFG/3AgR8AT3z3buBXplV5Fqm1pNrgYA4C0IQPBqs0cN0ej4CDW22PXyrkKzywEAeAkCELyaxWJxXRG27uNCnW7x7uVXAED/6NZVYAsWLLjg+zU1Ne7UAvTI9Zcn6Hdbjqqk6rT+uqdES65MNbskAICH69YMUFRU1AWPlJQU3XnnnX1VK9CpAJtV91ztnAVau+2YWu08jw4AcGHdmgHiEnd4qlszhmnVh1+ptOa0/vtAmRZMGmZ2SQAAD8YeIPiEkECbfjA9VZLz8RgOR6/d3QEA4IMIQPAZi6amKCI4QF9VNuh/v6g0uxwAgAcjAMFnRIYEatG0FEnSM9l56sV7fAIAfAwBCD7lB1cNV1CAVfuLa7TrWJXZ5QAAPBQBCD5lcESwbpvs3AC9Jiff5GoAAJ6KAASfc+/VI2WzWrTtyxM6VFprdjkAAA9EAILPSYoN07wJCZKYBQIAdI4ABJ+0/OzjMTYfLFfByUaTqwEAeBoCEHzS6PhIXTN6iByGtHYbs0AAgPYIQPBZ5x6SumFvqb6uO2NyNQAAT0IAgs+anBqrKamxarE7lPVRgdnlAAA8CAEIPu3cLNCru4pU29RqcjUAAE9BAIJPmzVqsEbHR6ixxa4XdxaaXQ4AwEMQgODTLBaLaxbohY8L1NTSZnJFAABPQACCz7v+8gQlx4apuqlVf/m0xOxyAAAegAAEnxdgs+remSMkSc9tO6ZWu8PkigAAZiMAwS/cPGmYBkcEq6z2jN7OLTO7HACAyQhA8AshgTbdPX24JOnZnHw5HIbJFQEAzEQAgt9YmJmsiJAA5VU26IMjX5tdDgDARAQg+I2IkEDdOS1FkvRMdr4Mg1kgAPBXBCD4lbuuGq7gAKsOlNRo57FTZpcDADAJAQh+ZdCAYH3viiRJ0ppsHpIKAP6KAAS/88MZI2SzWrT9q5M6eLzW7HIAACYgAMHvJMWG6Ya0REnSmpw8k6sBAJiBAAS/tHym8/EYmw9V6NiJBpOrAQD0NwIQ/NKo+AjNGTNEhiH9OeeY2eUAAPoZAQh+675Zl0iS3tp/XBW1Z0yuBgDQnwhA8FsZKTGaMjxWrXZDz29nFggA/AkBCH7t/lnOvUCv7S5WTVOLydUAAPoLAQh+beZlgzU2IVJNLXa9uKPI7HIAAP2EAAS/ZrFYdN/ZWaD1OwrU1NJmckUAgP5AAILfmzs+XikDw1Td1Ko3dpeYXQ4AoB8QgOD3AmxW3Xu1cxboue3H1NLmMLkiAEBfIwABkhZMGqrBEcEqrz2jt3NLzS4HANDHCECApJBAm5ZNHy5JejYnXw6HYXJFAIC+RAACzrojM1mRIQHKP9GoLYe/NrscAEAfMjUAbdu2TfPmzVNiYqIsFos2bdp0wfZLly6VxWLpcIwbN87V5rnnntOMGTMUExOjmJgYzZkzR7t37+7jnsAXRIQE6s5pqZKkNdl5MgxmgQDAV5kagBobG5WWlqbVq1d3qf2qVatUXl7uOkpKShQbG6tbb73V1SY7O1u33367tm7dqp07dyo5OVnXXnutSkvZ14GLu+uqVIUEWnXgeK125p8yuxwAQB+xGB7y11yLxaKNGzdq/vz5Xf7Mpk2btGDBAhUUFCglJaXTNna7XTExMVq9erXuvPPOLn1vXV2doqKiVFtbq8jIyC7XA9/w87cP6cWdRZp+ySC9sizT7HIAAF3Und/fXr0HKCsrS3PmzDlv+JGkpqYmtba2KjY29rxtmpubVVdX1+6A//rh1SNks1r0Ud5JfXa8xuxyAAB9wGsDUHl5uTZv3qxly5ZdsN2jjz6qoUOHas6cOedts3LlSkVFRbmOpKSk3i4XXmRYTJhuTEuUJK3Jzje5GgBAX/DaALR+/XpFR0dfcMnst7/9rV5//XW99dZbCgkJOW+7FStWqLa21nWUlHA3YH+3/OzjMd7/vEJ5lQ0mVwMA6G1eGYAMw9C6deu0ePFiBQUFddrmd7/7nZ5++mlt2bJFEyZMuOD3BQcHKzIyst0B/3ZZXITmjImTYUhrtzELBAC+xisDUE5OjvLy8nT33Xd3+v5//Md/6Je//KXef/99TZ48uZ+rg6+4f7ZzFmjj/lKV1542uRoAQG8yNQA1NDQoNzdXubm5kqSCggLl5uaquLhYknNpqrMrt7KyspSZmanx48d3eO+3v/2tHn/8ca1bt06pqamqqKhQRUWFGhpYxkD3TEqO0dQRsWq1G3p+e4HZ5QAAepGpAWjPnj1KT09Xenq6JOmRRx5Renq6nnzySUnOjc7nwtA5tbW12rBhw3lnf5555hm1tLTolltuUUJCguv43e9+17edgU+6b9YlkqTXdxerurHF5GoAAL3FY+4D5Em4DxDOMQxD//ZfH+nzsjo9NOdSPTTnMrNLAgCch9/cBwjoaxaLRfedvSJs/Y5CNTa3mVwRAKA3EICAi5g7PkGpA8NU09SqNz7lFgkA4AsIQMBF2KwW3TvTOQv0/PZjamlzmFwRAMBdBCCgCxZMGqohEcEqrz2jTft5sC4AeDsCENAFwQE2LZsxXJL07LZ82R1cOwAA3owABHTRHZkpigoN1LETjfrjB1+q9nSr2SUBAHqIAAR00YDgAP3gKucs0Oqtecp8+kP93zc/06HSWpMrAwB0F/cB6gT3AcL5OByGXv+0WC/vLNIXFfWu82lJ0Vo8NUX/NiFBIYE2EysEAP/Vnd/fBKBOEIBwMYZhaE9RtV7eWaTNh8rVanf+bxQdFqhbM4ZpYWaKUgeFm1wlAPgXApCbCEDojhP1zfrrnhK99kmxSmv++dDUqy8brEWZyfrW6CEKsLHaDAB9jQDkJgIQesLuMJR9tFIv7ypSzpcndO7/rMSoEN2RmazbrkjSkIgQc4sEAB9GAHITAQjuKjrVqNc+KdZf95Sousl5tViA1aLrxsdr8dQUTRkeK4vFYnKVAOBbCEBuIgCht5xpteu9g+V6eVeR9hfXuM5fFjdAi6am6Kb0oYoICTSvQADwIQQgNxGA0BcOldbq1U+KtGl/mU632iVJYUE23ZQ+VIumpmhMAv+tAYA7CEBuIgChL9WebtXGfcf18q4i5Z9odJ2fnBKjxdNSdN34eAUHcCk9AHQXAchNBCD0B8MwtPPYKb26q1h//7xCbWcfrzEwPEi3XZGkO6YkKyk2zOQqAcB7EIDcRABCf/u67oze2F2i13cXq6LujCTJYpG+NWqIFk1N0dWXDZbNyqZpALgQApCbCEAwS5vdoQ+PVOqVXUX6KO+k63xSbKgWZqbotslJig0PMrFCAPBcBCA3EYDgCY6daNCrnxTrb3tKVHemTZIUZLPq+gkJWjQ1WZOSY7iUHgD+BQHITQQgeJLTLXb994EyvbyrSAf/5cGrYxIitXhqim6cmKjw4AATKwQAz0AAchMBCJ7qQEmNXt5VpP8+UKbmNockKSI4QAsmOS+lvzQuwuQKAcA8BCA3EYDg6WqaWvTm3uN6ZVeRCk81uc5PHRGrRVNTdO3YeAUF8PwxAP6FAOQmAhC8hcNh6OP8k3p5Z5E+PPK1zl5Jr8ERwbr9iiTdnpmshKhQc4sEgH5CAHITAQjeqKzmtF7fXazXd5foZEOzJMlqkeaMidPiaSm6auQgWbmUHoAPIwC5iQAEb9bS5tCWwxV6eWeRPimocp0fPihcCzOTdUvGMEWHcSk9AN9DAHITAQi+4suv6/XqriJt2FeqhmbnpfTBAVbdkJaoRVNTlJYUbW6BANCLCEBuIgDB1zQ2t+nt3DK9tLNQX1TUu85PGBalRVNTNG9CokKDeP4YAO9GAHITAQi+yjAM7Suu1ss7i/TewQq12J2X0keGBOjWyUlamJmsEYMHmFwlAPQMAchNBCD4g1MNzfrrnuN69ZMiHa8+7To/49JBWpiZojljhijAxqX0ALwHAchNBCD4E7vD0LYvT+jlXUXaerRS5/5EiI8M0R2Zyfr+FUkaEhlibpEA0AUEIDcRgOCvSqqa9NruYv3l0xJVNbZIkgKsFn1nXLwWTk3WtBEDef4YAI9FAHITAQj+rrnNrvcPOS+l31NU7Tp/yZABWpSZrAUZwxQZEmhihQDQEQHITQQg4J8Ol9XplU+KtGl/qZpa7JKk0ECb5qc7L6UflxhlcoUA4EQAchMBCOio/kyrNu4v1cs7i/RVZYPr/KTkaC2amqLvXp6gkEAupQdgHgKQmwhAwPkZhqHdBVV6eVeR3j9UobazDyCLCQvUbVckaeGUFCUPDDO5SgD+iADkJgIQ0DWV9Wf0l90len13scpqz0iSLBZp5mWDtXhqimaNGiIbzx8D0E8IQG4iAAHd02Z36B9fVOrlXUXa/tVJ1/mh0aG6IzNZ37siSYMGBJtYIQB/QAByEwEI6LmCk4167ZMi/XXPcdWebpUkBdos+u7lCVo0NUWTU2K4lB5AnyAAuYkABLjvTKtd//NZuV7eVaQDJTWu86PjI7Roaormpw/VgOAA8woE4HMIQG4iAAG967PjNXplV5HeOVCmM63O548NCA7QTelDtWhqikbFR5hcIQBfQAByEwEI6Bu1Ta16c99xvbqrSMdONrrOTxkeq0VTU3TduHgFBfD8MQA9QwByEwEI6FuGYWhH/im9vLNIHxz5Wvazl9IPGhCk712RpDsyUzQ0OtTkKgF4GwKQmwhAQP+pqD2j13cX6/Xdxaqsb5YkWS3St0bHadHUZF196WBZuZQeQBcQgNxEAAL6X6vdoQ8Pf62XdxVpR/4p1/mUgWFamJmsWzOSFBMeZGKFADwdAchNBCDAXHmV9XplV7E27Duu+jNtkqSgAKv+bUKCFk9N0cSkaC6lB9ABAchNBCDAMzS1tOmd3DK9vKtIn5fVuc6PHxqpRZkpumFiosKCuJQegBMByE0EIMCzGIah3JIavbyrSP/zWbla2pyX0keEBOiWjGFamJmiS4YMMLlKAGYjALmJAAR4rqrGFv1tT4le/aRYxVVNrvNXjhyoxVNTNGdsnAJtXEoP+CMCkJsIQIDnczgMbfvqhF7ZVaR/fFGps1fSa0hEsG6fkqzbpyQrPirE3CIB9CsCkJsIQIB3OV7dpNd3F+svn5boZEOLJMlmtejasXFaNDVFV44cyKZpwA8QgNxEAAK8U0ubQ+9/XqFXdhZpd2GV6/yIweFalJmimzOGKSo00MQKAfQlApCbCECA9/uiok6v7CrSxn2lamyxS5JCAq26MW2ovj8lSROGRcvGDRYBn0IAchMBCPAdDc1t2ri/VK/uKtIXFfWu8xHBAZqYHK3JKbGanBqjiUnRCufp9IBXIwC5iQAE+B7DMLSnqFov7yzS/x752jUrdI7NatGYhAhlJMcoIzVWk1NilMjzyACvQgByEwEI8G1tdoe+qKjXvuJq7Sms1t6iapXWnO7QLjEqxBWGMlJiNDo+QgFcYg94LAKQmwhAgP8prz3tCkN7i6p1uLzO9ZT6c8KDbJqYHK2MFGcompgcrcgQNlUDnoIA5CYCEIDG5jYdKKnRnqJq7Smq1v6iatU3t7VrY7FIo+IiNDk1RpNTYpWREqNhMaFccg+YhADkJgIQgG+yOwx9VVnvmiXaU1SlkqqOy2ZxkcHKSIlxzRKNTYzkztRAPyEAuYkABKArKuvOnA1DzuPz0lq1fWPZLDTQprSkKGWkOGeJJiXHKCqMZTOgLxCA3EQAAtATp1vsOnC8xrWPaG9RtWpPt3Zod1ncANcMUUZKjFIGhrFsBvQCrwlA27Zt03/8x39o7969Ki8v18aNGzV//vzztl+6dKlefPHFDufHjh2rzz//3PV6w4YNeuKJJ5Sfn6+RI0fqV7/6lW666aYu10UAAtAbHA5D+ScatOdfAlHBycYO7QYNCFZGivOeRBmpMRqXGKngAJsJFQPerTu/v02961djY6PS0tJ011136eabb75o+1WrVunXv/6163VbW5vS0tJ06623us7t3LlT3/ve9/TLX/5SN910kzZu3KjbbrtNH330kTIzM/ukHwDQGavVokvjInRpXIRun5IsSTrZ0OwKQ3sKq3SotE4nG5r198+/1t8//1qSFBRgVdqwqHazRDHhQWZ2BfA5HrMEZrFYLjoD9E2bNm3SggULVFBQoJSUFEnS9773PdXV1Wnz5s2udtddd51iYmL0+uuvd/o9zc3Nam5udr2uq6tTUlISM0AA+tyZVrsOldY69xEVVmtfcbWqGls6tBsxOFyTz+4jykiN0YhB4SybAd/gNTNA7srKytKcOXNc4UdyzgA9/PDD7dp95zvf0Z/+9Kfzfs/KlSv1i1/8oq/KBIDzCgm0aXJqrCanxkoznXesLjjZ6Fw2K3RebZZ/olHHzh5/3XNckhQbHqRJyc7ZocmpMbp8aJRCAlk2A7rKawNQeXm5Nm/erNdee63d+YqKCsXFxbU7FxcXp4qKivN+14oVK/TII4+4Xp+bAQKA/maxWDRi8ACNGDxAt012/jlU3djivGv12VB04HiNqhpb9OGRr/XhkbPLZjarxg+N1ORU55Vmk1NjNGhAsJldATya1wag9evXKzo6utMls29OCxuGccGp4uDgYAUH8wcFAM8UEx6ka8bE6Zoxzr/ctbQ5dKisVvvOLpvtKarWyYZm7Suu0b7iGtfnUgeGOfcRpTpnii4ZPEBWK8tmgOSlAcgwDK1bt06LFy9WUFD7jYHx8fEdZnsqKys7zAoBgLcKCrBqUnKMJiXHaNkM55+JxVVNrjC0r6haX1bWq/BUkwpPNWnDPueyWVRooCYlR2tyqvOu1WnDohUaxLIZ/JNXBqCcnBzl5eXp7rvv7vDetGnT9MEHH7TbB7RlyxZdeeWV/VkiAPQbi8WilIHhShkYrpszhkmSak+3al+xc8lsb1G1cktqVHu6VVuPntDWoyckSQFWi8YlRrpmiSanxGhIZIiZXQH6jakBqKGhQXl5ea7XBQUFys3NVWxsrJKTk7VixQqVlpbqpZdeave5rKwsZWZmavz48R2+88EHH9TVV1+t3/zmN7rxxhv19ttv68MPP9RHH33U5/0BAE8RFRqo2aOGaPaoIZKkVrtDR8rr2j3K4+u6Zh04XqsDx2u17uMCSVJSbKjzjtUpzkB0WVyEbCybwQeZehl8dna2Zs+e3eH8kiVLtH79ei1dulSFhYXKzs52vVdbW6uEhAStWrVKP/zhDzv93jfffFOPP/64jh075roR4oIFC7pcFzdCBODrDMNQac3ps/cjci6dHa2o0zee5KGI4AClnw1DGSkxmpgUrfBgr1w8gB/wmjtBeyoCEAB/VH+mVbklNa5Zov3F1WpssbdrY7NaNCYhwnk/orOX4CdEhZpUMdAeAchNBCAAkNrsDn1RUe+8BP9sKCqtOd2hXWJUiDJS/3nX6tHxEQqwWU2oGP6OAOQmAhAAdK689rQrDO0tqtbh8jrZv7FuFh5k08TkaNejPNKToxUREmhSxfAnBCA3EYAAoGsam9t0oKTG+SiPomrtL6pWfXNbuzZWizQqPtI1Q5SREqNhMaE8ygO9jgDkJgIQAPSM3WHoq8r6dleblVR1XDaLiwx27SPKSInR2MRIBbJsBjcRgNxEAAKA3lNZd+ZsGHIen5fWqu0by2ahgTalJUW5HvY6KTlGUaEsm6F7CEBuIgABQN853WLXgeM1rn1Ee4uqVXu6tV0bi0W6dMgA1z6iyakxSo4NY9kMF0QAchMBCAD6j8NhKP9Eg/Nhr2ePgpONHdoNGhCsjJRo140axyVGKiSQR3ngnwhAbiIAAYC5TjY0u8LQnsIqHSqtU4vd0a5NoM2iMQmRmpgU7TpSB4bzwFc/RgByEwEIADzLmVa7DpXWOvcRFVYrt6RaJxtaOrSLDAlQWlK00pOiNTE5WhOTYhQbHtTJN8IXEYDcRAACAM9mGIaOV59WbkmN6zhUWqvmNkeHtsmxYf+cJUqO1tgEls58FQHITQQgAPA+rXaHjlbUa39JjXKLa5RbUq38Ex33EgXaLBqbEKm0f1k6Gz4onA3WPoAA5CYCEAD4htrTrfrs+LlA5DxONXZcOosKDXQFovSkaKUlRbN05oUIQG4iAAGAbzq3dLa/pEYHLrJ0ljIwrN0G67GJkQoOYOnMkxGA3EQAAgD/0Wp36IvyeuWWVDuXz0pqdOwCS2cT/2WDdepA7k3kSQhAbiIAAYB/q21q1YHjNe02WVd1snQWHRaotGH/3GA9cVi0Ylg6Mw0ByE0EIADAvzIMQyVVp7W/pFq5Z5fPDpXVqaWTpbPUs0tnaSyd9TsCkJsIQACAi2lpc+iLijrnDNHZTdbHOrmDdZDNqjGJkc57E509Ulg66xMEIDcRgAAAPVHb1Krc4/+8DD+3pEbVTa0d2sWEBba7DH9iUrSiw1g6cxcByE0EIABAbzAMQ8VVTe32En3eyWM9pH8unTn3E8VobEKkggKsJlTtvQhAbiIAAQD6SkubQ0fK69qFos4e/hpks2psovOqs/RkZzBKjmXp7EIIQG4iAAEA+lNNU0u7QHTgPEtnseFBShsWpYlJMUpLimLp7BsIQG4iAAEAzGQYhopOtV86O1zW+dLZ8EHh7fYSjfHjpTMCkJsIQAAAT9PcZteR8nrlFle7QlHhqaYO7YICrBp3dunM+WiPGCXFhvrF0hkByE0EIACAN6hubPmXq85qdOB4jWousnR27oaNUWGBJlTctwhAbiIAAQC80TeXzvaX1OhwWa1a7R1/1Y84t3R2doP16HjvXzojALmJAAQA8BXNbXYdLmt/1VnReZbOxidGujZYe+PSGQHITQQgAIAvq2ps0YGzM0TnrjqrPd1x6WxgeFC7GzamJUUrKtRzl84IQG4iAAEA/IlhGCo81eS8e/XZ/USHy+s6XzobHH52c3W0JibFaHRChAJtnrF0RgByEwEIAODvzrTadbi8zhWIcktqVFzVceks2HXVmXODdXpStIbFmLN0RgByEwEIAICOTjU068DZq872n106qzvT1qHdoAFBShsW7dpkPWFY/yydEYDcRAACAODiDMNQwcnGDjdsbHN0jBYjB4e3uwy/L5bOCEBuIgABANAzZ1rt+rzdVWfVKqk63aHd8EHh2vrTWb36s7vz+zugV38yAADwayGBNmWkxCgjJcZ17mRDsw78yyxRbkmNRsdHmFglAQgAAPSxQQOCdc2YOF0zJk6S5HAYamjpuHeoP3nGdWsAAMBvWK0WRYaYez8hAhAAAPA7BCAAAOB3CEAAAMDvEIAAAIDfIQABAAC/QwACAAB+hwAEAAD8DgEIAAD4HQIQAADwOwQgAADgdwhAAADA7xCAAACA3yEAAQAAvxNgdgGeyDAMSVJdXZ3JlQAAgK4693v73O/xCyEAdaK+vl6SlJSUZHIlAACgu+rr6xUVFXXBNhajKzHJzzgcDpWVlSkiIkIWi6VXv7uurk5JSUkqKSlRZGRkr363J/D1/km+30f65/18vY/0z/v1VR8Nw1B9fb0SExNltV54lw8zQJ2wWq0aNmxYn/6MyMhIn/0PW/L9/km+30f65/18vY/0z/v1RR8vNvNzDpugAQCA3yEAAQAAv0MA6mfBwcH6+c9/ruDgYLNL6RO+3j/J9/tI/7yfr/eR/nk/T+gjm6ABAIDfYQYIAAD4HQIQAADwOwQgAADgdwhAAADA7xCA+sAzzzyj4cOHKyQkRBkZGdq+ffsF2+fk5CgjI0MhISEaMWKEnn322X6qtGe607/s7GxZLJYOxxdffNGPFXfdtm3bNG/ePCUmJspisWjTpk0X/Yw3jV93++dt47dy5UpdccUVioiI0JAhQzR//nwdPXr0op/zpjHsSR+9aRzXrFmjCRMmuG6QN23aNG3evPmCn/Gm8etu/7xp7DqzcuVKWSwWPfTQQxdsZ8YYEoB62V/+8hc99NBDeuyxx7R//37NmDFDc+fOVXFxcaftCwoK9N3vflczZszQ/v379bOf/Uw/+clPtGHDhn6uvGu6279zjh49qvLyctdx6aWX9lPF3dPY2Ki0tDStXr26S+29bfy6279zvGX8cnJy9MADD2jXrl364IMP1NbWpmuvvVaNjY3n/Yy3jWFP+niON4zjsGHD9Otf/1p79uzRnj179K1vfUs33nijPv/8807be9v4dbd/53jD2H3Tp59+qrVr12rChAkXbGfaGBroVVOmTDGWL1/e7tzo0aONRx99tNP2/+f//B9j9OjR7c7de++9xtSpU/usRnd0t39bt241JBnV1dX9UF3vkmRs3Ljxgm28bfz+VVf6583jZxiGUVlZaUgycnJyztvGm8fQMLrWR28fx5iYGOP555/v9D1vHz/DuHD/vHXs6uvrjUsvvdT44IMPjJkzZxoPPvjgeduaNYbMAPWilpYW7d27V9dee22789dee6127NjR6Wd27tzZof13vvMd7dmzR62trX1Wa0/0pH/npKenKyEhQddcc422bt3al2X2K28aP3d46/jV1tZKkmJjY8/bxtvHsCt9PMfbxtFut+uNN95QY2Ojpk2b1mkbbx6/rvTvHG8buwceeEDXX3+95syZc9G2Zo0hAagXnTx5Una7XXFxce3Ox8XFqaKiotPPVFRUdNq+ra1NJ0+e7LNae6In/UtISNDatWu1YcMGvfXWWxo1apSuueYabdu2rT9K7nPeNH494c3jZxiGHnnkEU2fPl3jx48/bztvHsOu9tHbxvHgwYMaMGCAgoODtXz5cm3cuFFjx47ttK03jl93+udtYydJb7zxhvbt26eVK1d2qb1ZY8jT4PuAxWJp99owjA7nLta+s/Oeojv9GzVqlEaNGuV6PW3aNJWUlOh3v/udrr766j6ts7942/h1hzeP349+9CN99tln+uijjy7a1lvHsKt99LZxHDVqlHJzc1VTU6MNGzZoyZIlysnJOW9I8Lbx607/vG3sSkpK9OCDD2rLli0KCQnp8ufMGENmgHrRoEGDZLPZOsyGVFZWdki358THx3faPiAgQAMHDuyzWnuiJ/3rzNSpU/XVV1/1dnmm8Kbx6y3eMH4//vGP9c4772jr1q0aNmzYBdt66xh2p4+d8eRxDAoK0iWXXKLJkydr5cqVSktL06pVqzpt643j153+dcaTx27v3r2qrKxURkaGAgICFBAQoJycHP3nf/6nAgICZLfbO3zGrDEkAPWioKAgZWRk6IMPPmh3/oMPPtCVV17Z6WemTZvWof2WLVs0efJkBQYG9lmtPdGT/nVm//79SkhI6O3yTOFN49dbPHn8DMPQj370I7311lv6xz/+oeHDh1/0M942hj3pY2c8eRy/yTAMNTc3d/qet41fZy7Uv8548thdc801OnjwoHJzc13H5MmTtXDhQuXm5spms3X4jGlj2KdbrP3QG2+8YQQGBhpZWVnG4cOHjYceesgIDw83CgsLDcMwjEcffdRYvHixq/2xY8eMsLAw4+GHHzYOHz5sZGVlGYGBgcabb75pVhcuqLv9++Mf/2hs3LjR+PLLL41Dhw4Zjz76qCHJ2LBhg1lduKD6+npj//79xv79+w1Jxh/+8Adj//79RlFRkWEY3j9+3e2ft43ffffdZ0RFRRnZ2dlGeXm562hqanK18fYx7EkfvWkcV6xYYWzbts0oKCgwPvvsM+NnP/uZYbVajS1bthiG4f3j193+edPYnc83rwLzlDEkAPWB//f//p+RkpJiBAUFGZMmTWp3eeqSJUuMmTNntmufnZ1tpKenG0FBQUZqaqqxZs2afq64e7rTv9/85jfGyJEjjZCQECMmJsaYPn268e6775pQddecu+T0m8eSJUsMw/D+8etu/7xt/DrrmyTjhRdecLXx9jHsSR+9aRx/8IMfuP58GTx4sHHNNde4woFheP/4dbd/3jR25/PNAOQpY2gxjLM7jQAAAPwEe4AAAIDfIQABAAC/QwACAAB+hwAEAAD8DgEIAAD4HQIQAADwOwQgAADgdwhAAADA7xCAAKALLBaLNm3aZHYZAHoJAQiAx1u6dKksFkuH47rrrjO7NABeKsDsAgCgK6677jq98MIL7c4FBwebVA0Ab8cMEACvEBwcrPj4+HZHTEyMJOfy1Jo1azR37lyFhoZq+PDh+tvf/tbu8wcPHtS3vvUthYaGauDAgbrnnnvU0NDQrs26des0btw4BQcHKyEhQT/60Y/avX/y5EnddNNNCgsL06WXXqp33nmnbzsNoM8QgAD4hCeeeEI333yzDhw4oEWLFun222/XkSNHJElNTU267rrrFBMTo08//VR/+9vf9OGHH7YLOGvWrNEDDzyge+65RwcPHtQ777yjSy65pN3P+MUvfqHbbrtNn332mb773e9q4cKFqqqq6td+Auglff68eQBw05IlSwybzWaEh4e3O5566inDMAxDkrF8+fJ2n8nMzDTuu+8+wzAMY+3atUZMTIzR0NDgev/dd981rFarUVFRYRiGYSQmJhqPPfbYeWuQZDz++OOu1w0NDYbFYjE2b97ca/0E0H/YAwTAK8yePVtr1qxpdy42Ntb1z9OmTWv33rRp05SbmytJOnLkiNLS0hQeHu56/6qrrpLD4dDRo0dlsVhUVlama6655oI1TJgwwfXP4eHhioiIUGVlZU+7BMBEBCAAXiE8PLzDktTFWCwWSZJhGK5/7qxNaGhol74vMDCww2cdDke3agLgGdgDBMAn7Nq1q8Pr0aNHS5LGjh2r3NxcNTY2ut7/+OOPZbVaddlllykiIkKpqan63//9336tGYB5mAEC4BWam5tVUVHR7lxAQIAGDRokSfrb3/6myZMna/r06Xr11Ve1e/duZWVlSZIWLlyon//851qyZIn+/d//XSdOnNCPf/xjLV68WHFxcZKkf//3f9fy5cs1ZMgQzZ07V/X19fr444/14x//uH87CqBfEIAAeIX3339fCQkJ7c6NGjVKX3zxhSTnFVpvvPGG7r//fsXHx+vVV1/V2LFjJUlhYWH6+9//rgcffFBXXHGFwsLCdPPNN+sPf/iD67uWLFmiM2fO6I9//KN++tOfatCgQbrlllv6r4MA+pXFMAzD7CIAwB0Wi0UbN27U/PnzzS4FgJdgDxAAAPA7BCAAAOB32AMEwOuxkg+gu5gBAgAAfocABAAA/A4BCAAA+B0CEAAA8DsEIAAA4HcIQAAAwO8QgAAAgN8hAAEAAL/z/wOdKlFK1OiCCQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(trainer.get_hist_metric('fit'), label='fit')\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Customize your BP training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Actually, ``brainpy.train.BPTT`` is just one way to perform back-propagation training with your model. You can easily customize your training process.\n",
    "\n",
    "In the below, we demonstrate how to define a BP training process by hand with the above ANN model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:05:39.545832Z",
     "start_time": "2024-01-13T09:05:39.538563Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# packages we need\n",
    "\n",
    "from time import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:05:41.104484500Z",
     "start_time": "2024-01-13T09:05:39.959724100Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define the model\n",
    "model = ANNModel(28, 100, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:05:41.116952500Z",
     "start_time": "2024-01-13T09:05:41.107734700Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define the loss function\n",
    "def loss_fun(inputs, targets):\n",
    "  runner = bp.DSTrainer(model, progress_bar=False, numpy_mon_after_run=False)\n",
    "  predicts = runner.predict(inputs, reset_state=True)\n",
    "  predicts = bm.max(predicts, axis=1)\n",
    "  loss = bp.losses.cross_entropy_loss(predicts, targets)\n",
    "  acc = bm.mean(predicts.argmax(-1) == targets)\n",
    "  return loss, acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:05:41.783758700Z",
     "start_time": "2024-01-13T09:05:41.775583Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define the gradient function which computes the\n",
    "# gradients of the trainable weights\n",
    "grad_fun = bm.grad(loss_fun,\n",
    "                   grad_vars=model.train_vars().unique(),\n",
    "                   has_aux=True,\n",
    "                   return_value=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:05:42.802779100Z",
     "start_time": "2024-01-13T09:05:42.679333700Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define the optimizer we need\n",
    "opt = bp.optim.Adam(lr=1e-3, train_vars=model.train_vars().unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:05:43.129074800Z",
     "start_time": "2024-01-13T09:05:43.121707300Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# training function\n",
    "\n",
    "@bm.jit\n",
    "def train(xs, ys):\n",
    "  grads, loss, acc = grad_fun(xs, ys)\n",
    "  opt.update(grads)\n",
    "  return loss, acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-01-13T09:26:02.838665200Z",
     "start_time": "2024-01-13T09:05:43.623356100Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 100, Used 58.4698 s, Loss 1.0859, Acc 0.6189\n",
      "Step 200, Used 54.3465 s, Loss 0.5739, Acc 0.7942\n",
      "Step 300, Used 56.5062 s, Loss 0.5237, Acc 0.8098\n",
      "Step 400, Used 50.5268 s, Loss 0.4835, Acc 0.8253\n",
      "Step 500, Used 50.2707 s, Loss 0.4628, Acc 0.8318\n",
      "Step 600, Used 50.5184 s, Loss 0.4580, Acc 0.8305\n",
      "Step 700, Used 50.7511 s, Loss 0.4345, Acc 0.8420\n",
      "Step 800, Used 51.9514 s, Loss 0.4368, Acc 0.8414\n",
      "Step 900, Used 51.5502 s, Loss 0.4128, Acc 0.8491\n",
      "Step 1000, Used 51.4087 s, Loss 0.4140, Acc 0.8493\n",
      "Step 1100, Used 50.1260 s, Loss 0.4113, Acc 0.8484\n",
      "Step 1200, Used 50.2568 s, Loss 0.4038, Acc 0.8523\n",
      "Step 1300, Used 51.7090 s, Loss 0.3912, Acc 0.8555\n",
      "Step 1400, Used 51.2418 s, Loss 0.3937, Acc 0.8554\n",
      "Step 1500, Used 50.1411 s, Loss 0.3870, Acc 0.8577\n",
      "Step 1600, Used 50.4968 s, Loss 0.3765, Acc 0.8625\n",
      "Step 1700, Used 50.8128 s, Loss 0.3811, Acc 0.8599\n",
      "Step 1800, Used 52.4883 s, Loss 0.3744, Acc 0.8648\n",
      "Step 1900, Used 55.2034 s, Loss 0.3686, Acc 0.8652\n",
      "Step 2000, Used 51.4456 s, Loss 0.3738, Acc 0.8631\n",
      "Step 2100, Used 51.8214 s, Loss 0.3593, Acc 0.8697\n",
      "Step 2200, Used 50.2470 s, Loss 0.3571, Acc 0.8694\n",
      "Step 2300, Used 51.7452 s, Loss 0.3623, Acc 0.8680\n"
     ]
    }
   ],
   "source": [
    "# start training\n",
    "\n",
    "k = 0\n",
    "num_batch = 256\n",
    "running_loss = 0\n",
    "running_acc = 0\n",
    "print_step = 100\n",
    "X_train = bm.asarray(x_train)\n",
    "Y_train = bm.asarray(y_train)\n",
    "t0 = time()\n",
    "for _ in range(10):  # number of epoch\n",
    "  X_train = bm.random.permutation(X_train, key=123)\n",
    "  Y_train = bm.random.permutation(Y_train, key=123)\n",
    "\n",
    "  for i in range(0, X_train.shape[0], num_batch):\n",
    "    X = X_train[i: i + num_batch]\n",
    "    Y = Y_train[i: i + num_batch]\n",
    "    loss_, acc_ = train(X, Y)\n",
    "    running_loss += loss_\n",
    "    running_acc += acc_\n",
    "    k += 1\n",
    "    if k % print_step == 0:\n",
    "      print('Step {}, Used {:.4f} s, Loss {:0.4f}, Acc {:0.4f}'.format(\n",
    "        k, time() - t0,  running_loss / print_step, running_acc / print_step)\n",
    "      )\n",
    "      t0 = time()\n",
    "      running_loss = 0\n",
    "      running_acc = 0"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
